dfsummarizer package

Submodules

dfsummarizer.FlajoletMartin module

class dfsummarizer.FlajoletMartin.FMEstimator

Bases: object

estimate()

update(val)

update_all(vals)

dfsummarizer.FlajoletMartin.get_prefix(seed)

dfsummarizer.FlajoletMartin.trailing_zeros(n)

dfsummarizer.config module

dfsummarizer.dfsummarizer module

dfsummarizer.dfsummarizer: provides entry point main().

dfsummarizer.dfsummarizer.main()

dfsummarizer.dfsummarizer.print_usage(args)

Command line application usage instrutions.

dfsummarizer.funcs module

dfsummarizer.funcs.after_decimal(n)

dfsummarizer.funcs.analyse_df(df)

Given a pandas dataframe that is already in memory we generate a table of summary statistics and descriptors.

dfsummarizer.funcs.analyse_file(path_to_file)

dfsummarizer.funcs.analyse_file_in_chunks(path_to_file)

Given a path to a large dataset we will iteratively load it in chunks and build out the statistics necessary to summarise the whole dataset.

dfsummarizer.funcs.booleanize(x)

dfsummarizer.funcs.clean_dict(df, col)

dfsummarizer.funcs.coerce_dates(df)

dfsummarizer.funcs.combine_dicts(a, b, op=<built-in function add>)

dfsummarizer.funcs.count_lines(path_to_file)

Return a count of total lines in a file. In a way that filesize is irrelevant

dfsummarizer.funcs.extract_file_extension(path_to_file)

dfsummarizer.funcs.generate_final_summary(temp, total_chunks)

dfsummarizer.funcs.get_first_non_null_type(thecolumn, startpoint)

dfsummarizer.funcs.get_padded_number(n)

dfsummarizer.funcs.get_padded_val(val)

dfsummarizer.funcs.get_padded_val2(val, spacer)

dfsummarizer.funcs.get_percent_spacer(p)

dfsummarizer.funcs.get_spaces(spacer)

dfsummarizer.funcs.get_type_spacer(t)

dfsummarizer.funcs.infer_type(thetype, unicount, uniques)

dfsummarizer.funcs.infer_type_2(thecolumn, startpoint, unicount, uniques)

dfsummarizer.funcs.isNaN(num)

dfsummarizer.funcs.len_or_null(val)

Alternative len function that will simply return numpy.NA for invalid values This is need to get sensible results when running len over a column that may contain nulls

dfsummarizer.funcs.load_complete_dataframe(path_to_file)

We load the entire dataset into memory, using the file extension to determine the expected format. We are using encoding=’latin1’ because it ppears to permit loading of the largest variety of files. Representation of strings may not be perfect, but is not important for generating a summarization of the entire dataset.

dfsummarizer.funcs.print_csv(s)

dfsummarizer.funcs.print_latex(summary)

dfsummarizer.funcs.print_markdown(s)

dfsummarizer.funcs.round_down(n, decimals=0)

Round down a number to a specifed number of decimal places

dfsummarizer.funcs.update_temp_summary(temp, df, startpoint)

Module contents