Ag obair air an sgil a bhith a’ cleachdadh cruinneachadh agus fradharc dàta ann am Python

Hi Habr!

An-diugh obraichidh sinn air an sgil a bhith a’ cleachdadh innealan airson dàta a chruinneachadh agus fhaicinn ann am Python. Anns an solarachadh stòr-dàta air Github Dèanamaid sgrùdadh air grunn fheartan agus togaidh sinn seata de dhealbhan.

A rèir beul-aithris, aig an toiseach, mìnichidh sinn na h-amasan:

• Dàta buidhne a rèir gnè agus bliadhna agus sealladh air daineamaigs iomlan ìre breith an dà ghnè;
• Lorg na h-ainmean as mòr-chòrdte a-riamh;
• Roinn an ùine gu lèir san dàta gu 10 pàirtean agus airson gach fear, lorg an t-ainm as mòr-chòrdte airson gach gnè. Air son gach ainm a fhuaradh, seall air a dhiadhachd thar gach linn ;
• Airson gach bliadhna, obraich a-mach cia mheud ainm a tha a’ còmhdach 50% de dhaoine agus dèan sealladh (chì sinn na diofar ainmean airson gach bliadhna);
• Tagh 4 bliadhna bhon eadar-ama gu lèir agus seall airson gach bliadhna an cuairteachadh leis a’ chiad litir san ainm agus leis an litir mu dheireadh san ainm;
• Dèan liosta de ghrunn dhaoine ainmeil (ceann-suidhe, seinneadairean, cleasaichean, caractaran film) agus dèan measadh air a’ bhuaidh aca air daineamaigs ainmean. Tog sealladh lèirsinneach.

Nas lugha de fhaclan, barrachd còd!

Agus, rachamaid.

Nach cuir sinn an dàta ann am buidhnean a rèir gnè agus bliadhna agus dèanamaid sealladh air daineamaigs iomlan ìre breith an dà ghnè:

import numpy as np
import pandas as pd 
import matplotlib.pyplot as plt

years = np.arange(1880, 2011, 3)
datalist = 'https://raw.githubusercontent.com/wesm/pydata-book/2nd-edition/datasets/babynames/yob{year}.txt'
dataframes = []
for year in years:
    dataset = datalist.format(year=year)
    dataframe = pd.read_csv(dataset, names=['name', 'sex', 'count'])
    dataframes.append(dataframe.assign(year=year))

result = pd.concat(dataframes)
sex = result.groupby('sex')
births_men = sex.get_group('M').groupby('year', as_index=False)
births_women = sex.get_group('F').groupby('year', as_index=False)
births_men_list = births_men.aggregate(np.sum)['count'].tolist()
births_women_list = births_women.aggregate(np.sum)['count'].tolist()

fig, ax = plt.subplots()
fig.set_size_inches(25,15)

index = np.arange(len(years))
stolb1 = ax.bar(index, births_men_list, 0.4, color='c', label='Мужчины')
stolb2 = ax.bar(index + 0.4, births_women_list, 0.4, alpha=0.8, color='r', label='Женщины')

ax.set_title('Рождаемость по полу и годам')
ax.set_xlabel('Года')
ax.set_ylabel('Рождаемость')
ax.set_xticklabels(years)
ax.set_xticks(index + 0.4)
ax.legend(loc=9)

fig.tight_layout()
plt.show()

Lorgaidh sinn na h-ainmean as mòr-chòrdte ann an eachdraidh:

years = np.arange(1880, 2011)

dataframes = []
for year in years:
    dataset = datalist.format(year=year)
    dataframe = pd.read_csv(dataset, names=['name', 'sex', 'count'])
    dataframes.append(dataframe)

result = pd.concat(dataframes)
names = result.groupby('name', as_index=False).sum().sort_values('count', ascending=False)
names.head(10)

Roinnidh sinn an ùine gu lèir san dàta gu 10 pàirtean agus airson gach lorgaidh sinn an t-ainm as mòr-chòrdte airson gach gnè. Airson gach ainm a lorgar, bidh sinn a’ faicinn a daineamaigs fad na h-ùine:

years = np.arange(1880, 2011)
part_size = int((years[years.size - 1] - years[0]) / 10) + 1
parts = {}
def GetPart(year):
    return int((year - years[0]) / part_size)
for year in years:
    index = GetPart(year)
    r = years[0] + part_size * index, min(years[years.size - 1], years[0] + part_size * (index + 1))
    parts[index] = str(r[0]) + '-' + str(r[1])

dataframe_parts = []
dataframes = []
for year in years:
    dataset = datalist.format(year=year)
    dataframe = pd.read_csv(dataset, names=['name', 'sex', 'count'])
    dataframe_parts.append(dataframe.assign(years=parts[GetPart(year)]))
    dataframes.append(dataframe.assign(year=year))
    
result_parts = pd.concat(dataframe_parts)
result = pd.concat(dataframes)

result_parts_sums = result_parts.groupby(['years', 'sex', 'name'], as_index=False).sum()
result_parts_names = result_parts_sums.iloc[result_parts_sums.groupby(['years', 'sex'], as_index=False).apply(lambda x: x['count'].idxmax())]
result_sums = result.groupby(['year', 'sex', 'name'], as_index=False).sum()

for groupName, groupLabels in result_parts_names.groupby(['name', 'sex']).groups.items():
    group = result_sums.groupby(['name', 'sex']).get_group(groupName)
    fig, ax = plt.subplots(1, 1, figsize=(18,10))

    ax.set_xlabel('Года')
    ax.set_ylabel('Рождаемость')
    label = group['name']
    ax.plot(group['year'], group['count'], label=label.aggregate(np.max), color='b', ls='-')
    ax.legend(loc=9, fontsize=11)

    plt.show()

Airson gach bliadhna, bidh sinn a’ tomhas cia mheud ainm a tha a’ còmhdach 50% de dhaoine agus a’ faicinn an dàta seo:

dataframe = pd.DataFrame({'year': [], 'count': []})
years = np.arange(1880, 2011)
for year in years:
    dataset = datalist.format(year=year)
    csv = pd.read_csv(dataset, names=['name', 'sex', 'count'])
    names = csv.groupby('name', as_index=False).aggregate(np.sum)
    names['sum'] = names.sum()['count']
    names['percent'] = names['count'] / names['sum'] * 100
    names = names.sort_values(['percent'], ascending=False)
    names['cum_perc'] = names['percent'].cumsum()
    names_filtered = names[names['cum_perc'] <= 50]
    dataframe = dataframe.append(pd.DataFrame({'year': [year], 'count': [names_filtered.shape[0]]}))

fig, ax1 = plt.subplots(1, 1, figsize=(22,13))
ax1.set_xlabel('Года', fontsize = 12)
ax1.set_ylabel('Разнообразие имен', fontsize = 12)
ax1.plot(dataframe['year'], dataframe['count'], color='r', ls='-')
ax1.legend(loc=9, fontsize=12)

plt.show()

Taghamaid 4 bliadhna bhon eadar-ama gu lèir agus seallaidh sinn airson gach bliadhna an cuairteachadh leis a’ chiad litir san ainm agus leis an litir mu dheireadh san ainm:

from string import ascii_lowercase, ascii_uppercase

fig_first, ax_first = plt.subplots(1, 1, figsize=(14,10))
fig_last, ax_last = plt.subplots(1, 1, figsize=(14,10))

index = np.arange(len(ascii_uppercase))
years = [1944, 1978, 1991, 2003]
colors = ['r', 'g', 'b', 'y']
n = 0
for year in years:
    dataset = datalist.format(year=year)
    csv = pd.read_csv(dataset, names=['name', 'sex', 'count'])
    names = csv.groupby('name', as_index=False).aggregate(np.sum)
    count = names.shape[0]

    dataframe = pd.DataFrame({'letter': [], 'frequency_first': [], 'frequency_last': []})
    for letter in ascii_uppercase:
        countFirst = (names[names.name.str.startswith(letter)].count()['count'])
        countLast = (names[names.name.str.endswith(letter.lower())].count()['count'])

        dataframe = dataframe.append(pd.DataFrame({
            'letter': [letter],
            'frequency_first': [countFirst / count * 100],
            'frequency_last': [countLast / count * 100]}))

    ax_first.bar(index + 0.3 * n, dataframe['frequency_first'], 0.3, alpha=0.5, color=colors[n], label=year)
    ax_last.bar(index + bar_width * n, dataframe['frequency_last'], 0.3, alpha=0.5, color=colors[n], label=year)
    n += 1

ax_first.set_xlabel('Буква алфавита')
ax_first.set_ylabel('Частота, %')
ax_first.set_title('Первая буква в имени')
ax_first.set_xticks(index)
ax_first.set_xticklabels(ascii_uppercase)
ax_first.legend()

ax_last.set_xlabel('Буква алфавита')
ax_last.set_ylabel('Частота, %')
ax_last.set_title('Последняя буква в имени')
ax_last.set_xticks(index)
ax_last.set_xticklabels(ascii_uppercase)
ax_last.legend()

fig_first.tight_layout()
fig_last.tight_layout()

plt.show()

Dèan liosta de ghrunn dhaoine ainmeil (ceann-suidhe, seinneadairean, cleasaichean, caractaran film) agus measadh a dhèanamh air a ’bhuaidh aca air daineamaigs ainmean:

celebrities = {'Frank': 'M', 'Britney': 'F', 'Madonna': 'F', 'Bob': 'M'}
dataframes = []
for year in years:
    dataset = datalist.format(year=year)
    dataframe = pd.read_csv(dataset, names=['name', 'sex', 'count'])
    dataframes.append(dataframe.assign(year=year))

result = pd.concat(dataframes)

for celebrity, sex in celebrities.items():
    names = result[result.name == celebrity]
    dataframe = names[names.sex == sex]
    fig, ax = plt.subplots(1, 1, figsize=(16,8))

    ax.set_xlabel('Года', fontsize = 10)
    ax.set_ylabel('Рождаемость', fontsize = 10)
    ax.plot(dataframe['year'], dataframe['count'], label=celebrity, color='r', ls='-')
    ax.legend(loc=9, fontsize=12)
        
    plt.show()

Airson trèanadh, faodaidh tu ùine beatha an neach iomraiteach a chuir ris an t-sealladh bhon eisimpleir mu dheireadh gus a ’bhuaidh aca air daineamaigs ainmean a mheasadh gu soilleir.

Le seo, chaidh ar n-amasan uile a choileanadh agus a choileanadh. Tha sinn air an sgil a leasachadh ann a bhith a’ cleachdadh innealan airson dàta a chruinneachadh agus fhaicinn ann am Python, agus cumaidh sinn oirnn ag obair le dàta. Faodaidh a h-uile duine co-dhùnaidhean a dhèanamh stèidhichte air dàta deiseil, lèirsinneach iad fhèin.

Eòlas air a h-uile duine!

Source: www.habr.com