logo
Tags down

shadow

Efficient insertion of named tuple into pandas data frame


By : qnm01
Date : September 15 2020, 08:00 AM
With these it helps I have a named tuple called "rows" if I print it, it would look as shown below. How do I efficiently (not in for loop) insert this into a pandas data frame so that I can use it to plot graphs. (Number of rows and columns can go more than a thousand sometimes) , Why not just try:
code :
df = pd.DataFrame(rows)


Share : facebook icon twitter icon

Replace all None's in a Pandas data frame with a tuple of None's


By : Tuan Than
Date : March 29 2020, 07:55 AM
wish of those help One way to do it would be using pandas.DataFrame.apply() and pandas.Series.map() like this:
code :
df.apply(lambda ds: ds.map(lambda x: x if x != None else (None, None)))

How to create Pandas data frame from a tuple


By : pyrex.1337
Date : March 29 2020, 07:55 AM
should help you out The answer strongly depends on the String in your tuple. If what you copied is actually whats in the string, you have to convert the string to something pandas can parse, that's why I added the regex substitution.
code :
import pandas as pd
import io
import re
a = (('12','22','32'),
     """Column-1    Column-2    Column-3    colum-4 Column-5    Colum-6 Colum-7 Week    ACCT_YEAR   NAME
12  22  32  …   …   …   …   51  2016    Name-1
12  22  32  …   …   …   …   51  2016    Name-2
12  22  32  …   …   …   …   51  2016    Name-3""")
# The following substitution is only valid if there are absolutely no spaces in values
b = re.sub(string=a[1], pattern=' +', repl=',')
y = pd.read_csv(io.StringIO(b))
y

Subset pandas data frame based on tuple


By : Dan Baharir
Date : March 29 2020, 07:55 AM
I wish this helpful for you agg+isin
Since tuples are hashable, you can use isin and compare the aggregated values to your last. Using lst and a list directly instead of np.array helps.
code :
>>> lst = [('AA', 'P'), 
           ('BB', 'Q')]

>>> mask = df[['Firstnames', 'Lastnames']].agg(tuple, 1).isin(lst)
>>> df[mask]

    Firstnames  Lastnames   values
0   AA          P           10
1   BB          Q           13
3   AA          P           22
>>> df[mask].sort_values(by=['Firstnames', 'Lastnames'])

    Firstnames  Lastnames   values
0   AA          P           10
3   AA          P           22
1   BB          Q           13
>>> pd.concat([df[df.Firstnames.eq(a) & df.Lastnames.eq(b)] for a,b in lst])

    Firstnames  Lastnames   values
0   AA          P           10
3   AA          P           22
1   BB          Q           13
df = pd.concat([df]*10000).reset_index(drop=True)

%timeit mask = df[['Firstnames', 'Lastnames']].agg(tuple, 1).isin(lst); df[mask].sort_values(by=['Firstnames', 'Lastnames'])
942 ms ± 71.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit pd.concat([df[df.Firstnames.eq(a) & df.Lastnames.eq(b)] for a,b in lst])
16.2 ms ± 355 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
c = list(map(''.join, itertools.product(string.ascii_uppercase, string.ascii_uppercase)))
lst = [(a,b) for a,b in zip(c, list(string.ascii_uppercase)*26)]
df = pd.DataFrame({'Firstnames': c, 'Lastnames': list(string.ascii_uppercase)*26, 'values': 10})

%timeit mask = df[['Firstnames', 'Lastnames']].agg(tuple, 1).isin(lst); df[mask].sort_values(by=['Firstnames', 'Lastnames'])
15.1 ms ± 301 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit pd.concat([df[df.Firstnames.eq(a) & df.Lastnames.eq(b)] for a,b in lst])
781 ms ± 33.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Does pandas read the full data file and stores it in a data frame? Is it efficient to load a 100mb file in pandas?


By : user3015724
Date : March 29 2020, 07:55 AM
I hope this helps you . Personally I am using Pandas for files with sizes between some kilobytes and a few gigabytes without any problems. As stated here Pandas is very efficient with data 100MB up to 1GB. Thats pretty much what I observe when using Pandas.

Python loop through tuple list adding a value from pandas data frame


By : Chelomunoz7
Date : March 29 2020, 07:55 AM
around this issue I am trying to loop through a list of tuples adding a value to the end of each one that corresponds to a value in a column in a pandas data frame. , Use zip, list comprehension solution:
code :
df3 = [j + (i,) for i,j in zip(df2["Beta"], df1)]
for i,j in zip(df2["Beta"], df1):
    j = j + (i,)
    df3.append(j)
print(df3)
[(1, 2, 'a'), (4, 5, 'b'), (8, 9, 'c')] 
Related Posts Related Posts :
  • Unicodedata.normalize : TypeError: normalize() argument 2 must be str, not list
  • What uses the memory of my python process? (RSS vs VMS)
  • Pandas: filter dataframe by multiple conditions with column containing nan
  • Selenium webdriver.Remote driver does not work with tor proxy(webdriver.Chrome does)
  • generate keyword arguments from positional arguments in python
  • Find all words including those with special characters
  • The total maximum value of the value chosen
  • Nested while Loops is not working in python
  • Why do my python sklearn logistic regression results differ from this example?
  • Python Regex: How do I use regular expression to read in a file with multiple lines, and extract words from each line to
  • What is the use of drop_first in pandas?
  • Is it possible to pass a Flask session to another .py-File which is not in the routing?
  • tensorflow_hub to pull BERT embedding on windows machine - extending to albert
  • Python Pandas slicing with various datatypes
  • Pandas: Checking and changing all items in a column
  • Why does __call__ returned values get garbage collected when calling a class twice: SomeClass()()
  • Insert element at every nth location in list of lists
  • PD Read in Jupyter Notebook 3.7
  • Visualize Results of each iteration of While Loop into a Time Series Chart
  • Run a function for each row and create a new Column Pandas Dataframe
  • How can I create a small IDLE-like Python Shell in Tkinter?
  • extract variable and data from a string in python
  • CUDA implementation of Softmax
  • The function to_excel of pandas generate an unexpected TypeError
  • string is contain with newline symbol (\n), how to use regex to replace \n to \n?
  • How can I use %s to replace text within a file in python?
  • How to Reference a Pandas Column that has a dot in the name
  • How to use tuple as a key of a dictionary
  • How to extract two integer values from a column of a dataframe
  • How properly build a class in the __new__ with type(3 args) and 2 ancestors?
  • How to declare the return of a function as the default parameter to another function without calling the first function?
  • Elegant way to check arguments across multiple functions
  • How can I replace elemts of a list with other elements
  • i want to use variable globally in veiws.py
  • Pandas data not being plotted
  • Python Generator: How do I generate pairs from two different lists based on user input (of how many pairs to print)
  • Python: How to use a dictionary to call methods (values in dictionary) to run based on user input (key in dictionary) in
  • Read lines between two keywords Python
  • How do you insert data from the user into the file with the most optimal using Python?
  • How do you create a loop that will work in Snowflake?
  • Why can't I change the __class__ attribute of an instance of object?
  • Concatenating pandas dataframes from pickle vs. from in-memory dictionary - why does in-memory fail?
  • How to Calculate time difference between two date columns
  • In '<string>' requires string as left operand, not list
  • Django clean() change field requirement
  • Python - TypeError: write() argument must be str, not bytes
  • Commutative Count in a groupby dataframe on other columns condition
  • Undo np.fft.fft2 to get the original image
  • What is the proper way to share a program without sharing personal information?
  • Pandas DataFrame - summing rows by multiple column values
  • Python - best approach to mapping codes in data to description
  • I need to know how to do this, but it may be impossible
  • pandas dataframe columns with list values
  • Wrong value of standard deviation
  • Django POST error: tuple has no attribute get, despite similar code working previously
  • Convert a 3d-array into 2d-array
  • how to find the shortest distance between two regex
  • Python unit testing on class methods with no input arguments
  • Bokeh: Repeated plotting in jupyter lab increases (browser) memory usage
  • Cannot convert string into float in a for loop
  • shadow
    Privacy Policy - Terms - Contact Us © 35dp-dentalpractice.co.uk