Timeseries with overlapping timeframes, using just the most recent in Excel Power Query

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP



Timeseries with overlapping timeframes, using just the most recent in Excel Power Query



i have the following problem:



In Power Query i can bring together multiple csv files with power query, with the function to load from a folder. Unfortenatuly, the timeseries data we get is overlapping, with changed values for the entire timeseries.
So for example we get a file for the last 30 days every monday. The values in the last 30 days do change, so the values from my report last week might not be up to date anymore. My goal is to overwrite the old data from last week with the new data from this week.



Example:



File A


Date Item Hits Solved
01.01.2018 A 100 50
01.01.2018 B 138 65
02.01.2018 A 124 70
...
07.01.2018 A 143 50
07.01.2018 B 147 57
...
30.01.2018 A 250 124



File B


Date Item Hits Solved
07.07.2018 A 143 80
07.07.2018 B 147 95
...
06.02.2018 A 341 148



The data from file A should be updated with the new information from File B, keeping only the values from (in this case) the 01.01.2018 to the 06.01.2018 from file A. The rest should be from file B.



Until now i tried to give my files numbers (the newer the higher, so file A -> 1, file B -> 2), making a new (key) column with Date+Item, sorting (descendant) by file number and then removing duplicates in the key column. This sometimes works (only keeping the most recent entries), but sometimes it does not. Removing duplicates from the Date Column does not work for me. Here it just remove the newest entries, keeping the old.



Is there a better way to solve this problem?



Thanks alot!




2 Answers
2



Go the step where you are sorting the data based on your file order and put it inside Table.Buffer(), so it would look like


= Table.Buffer(Table.Sort(BLAH BLAH TABLE SORTING CODE))



I don't remember the technical reasons, but for purposes here this resets the "internal" sort order PQ uses when running remove duplicates to follow the visual sort order of the table.



Also it's not necessary to create a key column, you can just select your Item + Date Columns and remove duplicates and it will accomplish the same thing.





This works great. Thank you! Do i understand it correctly that PQ internally does not change/recalculate the order if you do not use Table.Buffer()?
– Arthur Pennt
Aug 13 at 9:06





I believe when you use remove duplicates the function removes them based on the original order the data was loaded in. Table.Buffer() reloads data from whatever you referenced, so the "load" order remove duplicates uses is reset as well. I'm assuming there is an internal key for data you don't see that PQ is using on the backend.
– Wedge
Aug 13 at 16:14



I would suggest starting with FileA and doing a left anti merge with FileB on the Date and Item columns. This will remove any rows from FileA that have matching Date and Item rows in FileB.


FileA


FileB


Date


Item


FileA


Date


Item


FileB



Then delete the new column that this merge created and append File B to the end and you should have the result you want.


File B





Thank you for your comment. This seems to work, but from my limited knowledge you can not merge files from a folder, can you? I thought you can just append files!?
– Arthur Pennt
Aug 13 at 9:08





There's no group merge, as far as I know, but you should still be able to use this approach by doing two folder loads. One where you filter out the most recent file (Query A) and one where you just pick the most recent file (Query B) and then treat Query A and Query B like FileA and FileB in my answer.
– Alexis Olson
Aug 13 at 13:49


FileA


FileB






By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

-

Popular posts from this blog

Firebase Auth - with Email and Password - Check user already registered

Image
Clash Royale CLAN TAG #URR8PPP Firebase Auth - with Email and Password - Check user already registered I want to check when a user attempts to signup with createUserWithEmailAndPassword() in Firebase user Authentication method, this user is already registered with my app. createUserWithEmailAndPassword() 8 Answers 8 To detect whether a user with that email address already exists, you can detect when the call to createUserWithEmailAndPassword () fails with auth/email-already-in-use . I see that @Srinivasan just posted an answer for this. createUserWithEmailAndPassword () auth/email-already-in-use Alternatively, you can detect that an email address is already used by calling fetchSignInMethodsForEmail() (previous called fetchProvidersForEmail() ). fetchSignInMethodsForEmail() fetchProvidersForEmail() When the user trying to create an user with same email address, the task response will be "Response: The email address is already in use by another account." mFirebas
Read more

Dynamically update html content plain JS

Image
Clash Royale CLAN TAG #URR8PPP Dynamically update html content plain JS So I have a script bound in my index.html, which literally has to fetch an array of jsons from an api, then dynamically update my page when get response. But as a result I'm getting an as the only update of my page. here is my script fetch(url) .then((resp) => resp.json()) .then((resp) => resp.results.forEach(item => fetch(item.url) .then((response)=> response.json()) .then((response) => pokeArr.push(response)) .catch(console.log) ) ).catch(console.log) const filler = () => if(!pokeArr) return 0 pokeArr.map((i,j)=> return `<tr>$i.name</tr>` ) const pokeindex = () => document.getElementById('a').appendChild(document.createElement(filler())) pokeindex() When I'm consoling it, I can see in the console all the responses I get, so I'm at least doing the fetching part right. What's parberakan ? What's pokeArr ? – T.J. Crowder Aug 8 at 12
Read more

How to determine optimal route across keyboard

Image
Clash Royale CLAN TAG #URR8PPP How to determine optimal route across keyboard I'm trying to automate the typing process in a video game. Players are given an on-screen QWERTY keyboard layout that they can navigate with the control stick (up, down, left, right) in order to select a letter to type. I was wondering if there was a module for Python (or some other resource I could explore) that would help my program find a path across the keyboard as it types each letter in a given message. In other words, if the first letter typed was "A" and the next letter was "B", the module would know that the program should move 1 space down on the keyboard and 4 spaces to the right. Currently, I have the program resetting to the "G" key after each letter is typed (allowing me to use fixed-directions as the program parses each character in the desired message). As you can imagine, this is quite inefficient and defeats the purpose of automating the typing in the fir
Read more