Speed up PostgreSQL to BigQuery

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP



Speed up PostgreSQL to BigQuery



I would like to upload some data that is currently stored in postGreSQL to Google Bigquery to see how the two tools compare.



To move data around there are many options but the most user friendly (for me) one I found thus far leverages the power of python pandas.


sql = "SELECT * FROM ".format(input_table_name)
i = 0
for chunk in pd.read_sql_query(sql , engine, chunksize=10000):
print("Chunk number: ",i)
i += 1
df.to_gbq(destination_table="my_new_dataset.test_pandas",
project_id = "aqueduct30",
if_exists= "append" )



however this approach is rather slow and I was wondering what options I have to speed things up. My table has 11 million rows and 100 columns.



The postGreSQL is on AWS RDS and I call python from an Amazon EC2 instance. Both are large and fast. I am currently not using multiple processors although there are 16 available.





We shift new data (daily partitions) daily from PG to Bigquery and for most cases we use program in golang. But for really huge tables like ~100 milions rows approx 20GB size we use bash script which wraps up export from PG into CSV format and load to BQ using bq command line tools. Because with golang program it took several hours. Script covers also conversion of table structure into data types used on BQ - postgresql.freeideas.cz/…
– JosMac
Jul 10 at 13:26



bq




1 Answer
1



As alluded to by the comment from JosMac, your solution/approach simply won't scale with large datasets. As you're already running on AWS/RDS then something like the following would be better in my opinion:





great sugestion. For which part do I need cloud functions and for which part dataflow. I've not used those services and the field is crowded with pub/sub aws lambda etc.
– Rutger Hofste
Jul 11 at 9:01





shinesolutions.com/2017/03/23/…
– Graham Polley
Jul 11 at 10:11





Eventually I've used a large EC2 instance with 32 vCPUs to query the database and send the resulting csv files directly to Google Cloud Storage, skipping the S3 step.
– Rutger Hofste
Jul 17 at 12:29






By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

-

Popular posts from this blog

Firebase Auth - with Email and Password - Check user already registered

Image
Clash Royale CLAN TAG #URR8PPP Firebase Auth - with Email and Password - Check user already registered I want to check when a user attempts to signup with createUserWithEmailAndPassword() in Firebase user Authentication method, this user is already registered with my app. createUserWithEmailAndPassword() 8 Answers 8 To detect whether a user with that email address already exists, you can detect when the call to createUserWithEmailAndPassword () fails with auth/email-already-in-use . I see that @Srinivasan just posted an answer for this. createUserWithEmailAndPassword () auth/email-already-in-use Alternatively, you can detect that an email address is already used by calling fetchSignInMethodsForEmail() (previous called fetchProvidersForEmail() ). fetchSignInMethodsForEmail() fetchProvidersForEmail() When the user trying to create an user with same email address, the task response will be "Response: The email address is already in use by another account." mFirebas
Read more

Dynamically update html content plain JS

Image
Clash Royale CLAN TAG #URR8PPP Dynamically update html content plain JS So I have a script bound in my index.html, which literally has to fetch an array of jsons from an api, then dynamically update my page when get response. But as a result I'm getting an as the only update of my page. here is my script fetch(url) .then((resp) => resp.json()) .then((resp) => resp.results.forEach(item => fetch(item.url) .then((response)=> response.json()) .then((response) => pokeArr.push(response)) .catch(console.log) ) ).catch(console.log) const filler = () => if(!pokeArr) return 0 pokeArr.map((i,j)=> return `<tr>$i.name</tr>` ) const pokeindex = () => document.getElementById('a').appendChild(document.createElement(filler())) pokeindex() When I'm consoling it, I can see in the console all the responses I get, so I'm at least doing the fetching part right. What's parberakan ? What's pokeArr ? – T.J. Crowder Aug 8 at 12
Read more

Creating a leaderboard in HTML/JS

Image
Clash Royale CLAN TAG #URR8PPP Creating a leaderboard in HTML/JS I'm trying to create a top 10 leaderboard, for now, I've only implemented 3 players, but I'm coming across an issue. My script doesn't seem to be outputting anything on webpage. The purpose of the script is to compare 3 arrays, player1,player2,and player3 solely based off of score. Which is the 2nd index of each array. I want the script to list the players in order of highest score. This method doesn't really seem to be efficient, especially when I'll have to add 7 more players. Could someone suggest perhaps an easier approach, as well as show me why my script wont output the info. Here's the JS function leaderboard() { var player1 = name:"Thomas",date:"01/23/18",score:"201"; var player2 = name:"Michael",date:"03/24/17",score:"943"; var player3 = Name:"Lisa",date:"06/04/18",Score:"79"; if(player1[2]&g
Read more