Data Scientology

2022-05-29 12:14:51 I want to convert a large JSON file into Tabular Format.

How should I approach this problem? The deadline is of 5-6 days from now so they definitely want me to go into a lot of details of possible solutions for this problem.

Pandas.jsonnormalize does help in converting list of dictionaries into Pandas dataframe but it's taking a lot of time for big file (512 MB .log file).

Should I try different sizes of Chunks of the dataset using jsonnormalize and see how much time does it take? Or should I be approaching this problem in a completely different way?

Edit: I'm currently dealing with a 512 MB file, but the test files would range from 4 GB to 48 GB.

/r/datascience
https://redd.it/uzn6n2 Open / Comment

2022-05-29 11:14:53 Deadliest Wars in Europe Since World War 2 [OC]

/r/dataisbeautiful
https://redd.it/v021hm Open / Comment

2022-05-29 10:14:56 Data_irl

/r/data_irl
https://redd.it/uzmctz Open / Comment

2022-05-29 08:14:44 Futureproofing job as we move from Excel to databases.

I work for a natural resources company which mostly manages its data through Excel, of which I am a heavy user. Its been mentioned that our Excel dependency is unsustainable and we need to move to databases instead.

This move is probably still a few years away but what can I do in the meantime to prepare? I am intermediate/advanced excel user but have zero experience with creating/using/accessing databases.

Will training be necessary or can be mostly self taught? Can I just start leaning SQL or are there other steps to do first, or other languages to consider?

/r/datascience
https://redd.it/uzwove Open / Comment

2022-05-29 07:14:46 [OC] Estimates of interprovincial migrants by province of origin and destination, Canada, 2020-2021

/r/dataisbeautiful
https://redd.it/v0168f Open / Comment

2022-05-29 06:14:55 [OC] Liverpool and Real Madrid's paths through the knock out stages to the Champions League final

/r/dataisbeautiful
https://redd.it/uzjhva Open / Comment

2022-05-29 05:14:51 US Percentage of Yearly Median Wage Going To Yearly Median Rent [OC]

/r/dataisbeautiful
https://redd.it/v008cb Open / Comment

2022-05-29 04:15:00 Percent of electricity generated from renewable sources across the US and the EU. Renewable sources include hydro, solar, wind, geothermal, and biomass. Nuclear is not counted as renewable in this comparison [OC]

/r/dataisbeautiful
https://redd.it/uzriuk Open / Comment

2022-05-29 03:14:48 Countries considered as electoral democracies by American organization freedom house. 2012 and 2022

/r/MapPorn
https://redd.it/uzlmcn Open / Comment

2022-05-29 02:15:09 Open / Comment