Merge rows that have time overlapping - Pandas

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP



Merge rows that have time overlapping - Pandas



Supposedly, I have dataframe like below.


Date Time_Start Time_End Reason
0 2018-08-05 2018-08-05 10:10:00 2018-08-05 13:35:00 blah1
1 2018-08-05 2018-08-05 12:50:00 2018-08-05 14:26:00 blah2
2 2018-08-05 2018-08-05 16:40:00 2018-08-05 17:30:00 blah3
3 2018-08-05 2018-08-05 17:00:00 2018-08-05 17:10:00 blah4
4 2018-08-06 2018-08-06 09:40:00 2018-08-06 11:23:00 blah5



I would like to know if pandas is capable of merging rows that have time overlapped e.g. take "Time_Start" of row one and merge with "Time_End" of second row and append the reason to a list. Another case is to remove the forth row as it is a sub time window of the third row and appending the reasons to a list. So, my expected output would look like this


Date Time_Start Time_End Reason
0 2018-08-05 2018-08-05 10:10:00 2018-08-05 14:26:00 [blah1, blah2]
1 2018-08-05 2018-08-05 16:40:00 2018-08-05 17:30:00 [blah3, blah4]
3 2018-08-06 2018-08-06 09:40:00 2018-08-06 11:23:00 [blah5]



Note that there can be more than two rows of data which have time overlapping.




1 Answer
1



You can create a help key here


df['newid']=(df['Time_Start']-df['Time_End'].shift()).dt.total_seconds().gt(0).cumsum()

newdf=df.groupby(['Date','newid']).agg('Time_Start':'min','Time_End':'max','Reason':lambda x : [x.values.tolist()]).reset_index(level=1)
newdf.assign(Reason=newdf.Reason.str[0])
Out[592]:
newid Time_Start Reason Time_End
Date
2018-08-05 0 2018-08-05 10:10:00 [blah1, blah2] 2018-08-05 14:26:00
2018-08-05 1 2018-08-05 16:40:00 [blah3, blah4] 2018-08-05 17:30:00
2018-08-06 2 2018-08-06 09:40:00 [blah5] 2018-08-06 11:23:00





Thank you for your attempt. It looks very promising. However, the "Time_End" of the second row in the result dataframe should be "2018-08-05 17:30:00".
– bensw
Aug 8 at 4:18





@bensw updated :-)
– Wen
Aug 8 at 4:20





Very splendid. You are the fast man alive. Thank you ;)
– bensw
Aug 8 at 4:24







By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

-

Popular posts from this blog

make 2 or more post in bootsrap

Image
Clash Royale CLAN TAG #URR8PPP make 2 or more post in bootsrap I am making a CMS and I want to add two or more content post in every row, I can made one post in one columns, please see my code below. I am using PHP , HTML and bootstrap . CMS PHP HTML bootstrap How can I add two or more content post in every row? Code: <div class="container"> <div class="row"> <!-- Post Content Column --> <div class="col-lg-4 border border-dark rounded"> <?php $query = "SELECT * FROM POSTS"; $run_query_post = mysqli_query($connection, $query); while ($row = mysqli_fetch_assoc($run_query_post)) $post_id = $row['post_id']; $post_title = $row['post_title']; $post_author = $row['post_author']; $post_date = $row['post_date']; $post_image = $row['post_image']; $post_content = substr($row['post_content'], 0, 50); $post_status = $row['post_status']; if ($post_status !=...
Read more

Store custom data using WC_Cart add_to_cart() method in Woocommerce 3

Image
Clash Royale CLAN TAG #URR8PPP Store custom data using WC_Cart add_to_cart() method in Woocommerce 3 I am creating a membership site and totally created static pages for each Membership plans (have only 3 plans). However, I have added products for each plan and when I hit SELECT PLAN button I redirect to some custom form where I ask users range of info we are going to use to fulfil the plan (same as sneakertub.com). I have written code into the PHP page which will handle SUBMIT action of the form. This PHP file, infopage.php , will process POST data I sent via POST call and stores these all data into WC session. infopage.php $customer_name = $_POST["customer_name"]; $customer_email = $_POST["customer_email"]; $customer_sex = $_POST["customer_sex"]; $customer_age = $_POST["customer_age"]; $product_id = $_POST["product_id"]; global $wp_session; $data = array( 'customer_name' => $customer_name, 'customer_email' =...
Read more

Firebase Auth - with Email and Password - Check user already registered

Image
Clash Royale CLAN TAG #URR8PPP Firebase Auth - with Email and Password - Check user already registered I want to check when a user attempts to signup with createUserWithEmailAndPassword() in Firebase user Authentication method, this user is already registered with my app. createUserWithEmailAndPassword() 8 Answers 8 To detect whether a user with that email address already exists, you can detect when the call to createUserWithEmailAndPassword () fails with auth/email-already-in-use . I see that @Srinivasan just posted an answer for this. createUserWithEmailAndPassword () auth/email-already-in-use Alternatively, you can detect that an email address is already used by calling fetchSignInMethodsForEmail() (previous called fetchProvidersForEmail() ). fetchSignInMethodsForEmail() fetchProvidersForEmail() When the user trying to create an user with same email address, the task response will be "Response: The email address is already in use by another account." mFirebas...
Read more