Ukuxazulula isibalo sokuhlehla komugqa okulula

I-athikili idingida izindlela ezimbalwa zokunquma isibalo sezibalo somugqa wokuhlehla olula (obhanqiwe).

Zonke izindlela zokuxazulula isibalo okuxoxwe ngaso lapha zisekelwe endleleni yezikwele ezincane. Ake sibonise izindlela ezilandelayo:

• Isixazululo sokuhlaziya
• Ukwehla kweGradient
• Ukwehla kwe-Stochastic gradient

Ngendlela ngayinye yokuxazulula i-equation yomugqa oqondile, i-athikili ihlinzeka ngemisebenzi ehlukahlukene, ehlukaniswe ikakhulukazi leyo ebhaliwe ngaphandle kokusebenzisa umtapo wezincwadi. I-NumPy nalabo abasebenzisa izibalo I-NumPy. Kukholelwa ukuthi ukusetshenziswa okunekhono I-NumPy izokwehlisa izindleko zekhompyutha.

Yonke ikhodi enikezwe esihlokweni ibhalwe ngolimi i-python 2.7 usebenzisa Incwadi kaJupyter. Ikhodi yomthombo nefayela elinesampula yedatha kuthunyelwa kuyo Github

Lesi sihloko sihloselwe kakhulu kokubili abaqalayo nalabo asebevele baqala kancane kancane ukufunda isifundo sesigaba esibanzi kakhulu kubuhlakani bokufakelwa - ukufunda ngomshini.

Ukuze sicacise indaba, sisebenzisa isibonelo esilula kakhulu.

Izimo zesibonelo

Sinamanani amahlanu abonisa ukuncika Y kusukela X (Ithebula No. 1):

Ithebula No. 1 “Izimo zesibonelo”

Sizocabanga ukuthi amanani yinyanga yonyaka, futhi - imali engenayo kule nyanga. Ngamanye amazwi, imali engenayo incike enyangeni yonyaka, futhi - uphawu kuphela imali engenayo encike kulo.

Isibonelo sinjalo, kokubili ngokombono wokuncika okunemibandela kwemali engenayo enyangeni yonyaka, futhi ngokombono wenombolo yamanani - kukhona ambalwa kakhulu awo. Kodwa-ke, ukwenza lula okunjalo kuzokwenza kube lula, njengoba besho, ukuchaza, hhayi kalula, ukwaziswa abaqalayo abakutholayo. Futhi nokulula kwezinombolo kuzovumela labo abafisa ukuxazulula isibonelo ephepheni ngaphandle kwezindleko ezibalulekile zomsebenzi.

Ake sicabange ukuthi ukuncika okunikezwe esibonelweni kungalinganiselwa kahle ngezibalo zezibalo zomugqa wokuhlehla olula (obhanqiwe) wefomu:

kuphi inyanga okwatholwa ngayo imali engenayo, - imali engenayo ehambisana nenyanga, и ama-coefficients okwehla komugqa olinganiselwe.

Qaphela ukuthi i-coefficient ngokuvamile ebizwa ngokuthi umthambeka noma ukuthambeka komugqa olinganiselwe; imele inani i uma ishintsha .

Ngokusobala, umsebenzi wethu esibonelweni ukukhetha ama-coefficient anjalo ku-equation и , lapho ukuchezuka kwamanani ethu emali engenayo abaliwe ngenyanga kusuka ezimpendulweni zangempela, i.e. amanani ethulwa kusampula azoba mancane.

Indlela yesikwele encane

Ngokwendlela yezikwele ezincane, ukuchezuka kufanele kubalwe ngokusikwela. Le nqubo ikuvumela ukuthi ugweme ukwesulwa okufanayo kokuphambuka uma kunezimpawu eziphambene. Isibonelo, uma kwesinye isimo, ukuchezuka +5 (kanye nesihlanu), kanti kwenye -5 (khipha okuhlanu), bese isamba sokuchezuka sizokhansela ngaphandle futhi sibe ngu-0 (ziro). Kungenzeka ukuthi ungasikwele ukuchezuka, kodwa ukusebenzisa indawo yemoduli bese konke ukuchezuka kuzoba kuhle futhi kuzonqwabelana. Ngeke sigxile kuleli phuzu ngokuningiliziwe, kodwa simane sibonise ukuthi ukuze kube lula ukubala, kuwumkhuba ukufaka isikwele ukuphambuka.

Lena yindlela ifomula ebukeka ngayo esizonquma ngayo isamba esincane sokuchezuka okuyisikwele (amaphutha):

kuphi kuwumsebenzi wokulinganisa izimpendulo zangempela (okungukuthi, imali engenayo esiyibalile),

yizimpendulo eziyiqiniso (imali engenayo inikezwe kusampula),

inkomba yesampula (inombolo yenyanga lapho kunqunywa khona ukuchezuka)

Masihlukanise umsebenzi, sichaze izilinganiso zokuhlukanisa ingxenye, futhi silungele ukudlulela kusixazululo sokuhlaziya. Kodwa okokuqala, ake sithathe uhambo olufushane mayelana nokuthi kuyini ukuhlukaniswa futhi sikhumbule incazelo yejometri yokuphuma kokunye.

Umehluko

Umehluko umsebenzi wokuthola okuphuma kokunye komsebenzi.

I-derivative isetshenziselwa ini? Okuphuma kokunye komsebenzi kubonakalisa izinga lokushintsha komsebenzi futhi kusitshela indlela yawo. Uma okuphuma kokunye endaweni ethile kuyiphozithivu, khona-ke umsebenzi uyakhula; ngaphandle kwalokho, umsebenzi uyancipha. Futhi uma likhulu inani lokuphuma kokunye okuphelele, izinga lokushintsha lamanani omsebenzi liyaphakama, kanye nokukhuphuka komthambeka wegrafu yomsebenzi.

Isibonelo, ngaphansi kwemibandela yesistimu yokuxhumanisa i-Cartesian, inani lokuphuma kokunye endaweni engu-M(0,0) lilingana no- + 25 kusho ukuthi endaweni ethile, lapho inani ligudluzwa kwesokudla ngeyunithi evamile, inani inyuka ngamayunithi angama-25 ajwayelekile. Kugrafu kubukeka sengathi amanani akhuphuka kancane kusuka endaweni ethile.

Esinye isibonelo. Inani lokuphuma kokunye liyalingana -0,1 kusho ukuthi uma esuswa ngeyunithi eyodwa evamile, inani yehla ngo-0,1 kuphela iyunithi evamile. Ngesikhathi esifanayo, kugrafu yomsebenzi, singakwazi ukubona umthambeka owehlayo ongabonakali kahle. Ukudweba isifaniso nentaba, kunjengokungathi sehla kancane kancane emthambekeni omnene usuka entabeni, ngokungafani nesibonelo sangaphambilini, lapho kudingeke khona ukugibela iziqongo eziphakeme kakhulu :)

Ngakho, ngemva kokuhlukanisa umsebenzi ngamaphutha и , sichaza izilinganiso ezihlukanisayo ze-oda loku-1. Ngemuva kokunquma izilinganiso, sizothola uhlelo lwezibalo ezimbili, ngokuxazulula esizokwazi ukukhetha amanani anjalo wama-coefficients. и , lapho amanani okuphuma kokunye okuhambisanayo emaphuzwini athile ashintsha khona ngenani elincane kakhulu, futhi esimweni sesixazululo sokuhlaziya awashintshi nhlobo. Ngamanye amazwi, umsebenzi wephutha kuma-coefficients atholiwe uzofinyelela ubuncane, njengoba amanani okuphuma okuyingxenye kulawa maphuzu azolingana noziro.

Ngakho-ke, ngokwemithetho yokuhlukanisa, i-equation eyingxenye ye-oda loku-1 ngokuphathelene ne-coefficient. izothatha ifomu:

I-oda loku-1 i-equation yengxenye ephuma kokunye ngokuphathelene nokuthi izothatha ifomu:

Ngenxa yalokho, sithole isistimu yezibalo enesixazululo sokuhlaziya esilula:

qala{equation*}
qala{cases}
na + bsumlimits_{i=1}^nx_i — sumlimits_{i=1}^ny_i = 0

sumlimits_{i=1}^nx_i(a +bsumlimits_{i=1}^nx_i - sumlimits_{i=1}^ny_i) = 0
end{cases}
ukuphela{isibalo*}

Ngaphambi kokuxazulula isibalo, masilayishe kuqala, sihlole ukuthi ukulayisha kulungile, bese sifometha idatha.

Ilayisha futhi ifometha idatha

Kufanele kuqashelwe ukuthi ngenxa yeqiniso lokuthi ngesixazululo sokuhlaziya, futhi kamuva ukwehla kwe-gradient kanye ne-stochastic gradient, sizosebenzisa ikhodi ngokuhlukahluka okubili: sisebenzisa umtapo wolwazi. I-NumPy futhi ngaphandle kokuyisebenzisa, sizobe sesidinga ukufometha kwedatha okufanele (bona ikhodi).

Ikhodi yokulayisha nokucubungula idatha

# импортируем все нужные нам библиотеки
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import math
import pylab as pl
import random

# графики отобразим в Jupyter
%matplotlib inline

# укажем размер графиков
from pylab import rcParams
rcParams['figure.figsize'] = 12, 6

# отключим предупреждения Anaconda
import warnings
warnings.simplefilter('ignore')

# загрузим значения
table_zero = pd.read_csv('data_example.txt', header=0, sep='t')

# посмотрим информацию о таблице и на саму таблицу
print table_zero.info()
print '********************************************'
print table_zero
print '********************************************'

# подготовим данные без использования NumPy

x_us = []
[x_us.append(float(i)) for i in table_zero['x']]
print x_us
print type(x_us)
print '********************************************'

y_us = []
[y_us.append(float(i)) for i in table_zero['y']]
print y_us
print type(y_us)
print '********************************************'

# подготовим данные с использованием NumPy

x_np = table_zero[['x']].values
print x_np
print type(x_np)
print x_np.shape
print '********************************************'

y_np = table_zero[['y']].values
print y_np
print type(y_np)
print y_np.shape
print '********************************************'

Ukubonakala

Manje, ngemva kokuba, okokuqala, silayishe idatha, okwesibili, sahlola ukufaneleka kokulayisha futhi ekugcineni safometha idatha, sizokwenza ukubonwa kokuqala. Indlela evame ukusetshenziselwa lokhu isiqephu imitapo yolwazi ozalwa olwandle. Esibonelweni sethu, ngenxa yezinombolo ezilinganiselwe, asikho isidingo sokusebenzisa umtapo wolwazi ozalwa olwandle. Sizosebenzisa umtapo wolwazi ojwayelekile I-Matplotlib futhi bheka nje uhlaka.

Ikhodi ye-scatterplot

print 'График №1 "Зависимость выручки от месяца года"'

plt.plot(x_us,y_us,'o',color='green',markersize=16)
plt.xlabel('$Months$', size=16)
plt.ylabel('$Sales$', size=16)
plt.show()

Ishadi No. 1 “Ukuncika emalini engenayo enyangeni yonyaka”

Isixazululo sokuhlaziya

Masisebenzise amathuluzi avame kakhulu ku i-python futhi uxazulule isistimu yezibalo:

qala{equation*}
qala{cases}
na + bsumlimits_{i=1}^nx_i — sumlimits_{i=1}^ny_i = 0

sumlimits_{i=1}^nx_i(a +bsumlimits_{i=1}^nx_i - sumlimits_{i=1}^ny_i) = 0
end{cases}
ukuphela{isibalo*}

Ngokomthetho kaCramer sizothola i-determinant jikelele, kanye nezinqumo ngo futhi , ngemva kwalokho, ukuhlukanisa isihlukanisi nge kusihlukanisi esivamile - thola i-coefficient , ngokufanayo sithola i-coefficient .

Ikhodi yesixazululo sokuhlaziya

# определим функцию для расчета коэффициентов a и b по правилу Крамера
def Kramer_method (x,y):
        # сумма значений (все месяца)
    sx = sum(x)
        # сумма истинных ответов (выручка за весь период)
    sy = sum(y)
        # сумма произведения значений на истинные ответы
    list_xy = []
    [list_xy.append(x[i]*y[i]) for i in range(len(x))]
    sxy = sum(list_xy)
        # сумма квадратов значений
    list_x_sq = []
    [list_x_sq.append(x[i]**2) for i in range(len(x))]
    sx_sq = sum(list_x_sq)
        # количество значений
    n = len(x)
        # общий определитель
    det = sx_sq*n - sx*sx
        # определитель по a
    det_a = sx_sq*sy - sx*sxy
        # искомый параметр a
    a = (det_a / det)
        # определитель по b
    det_b = sxy*n - sy*sx
        # искомый параметр b
    b = (det_b / det)
        # контрольные значения (прооверка)
    check1 = (n*b + a*sx - sy)
    check2 = (b*sx + a*sx_sq - sxy)
    return [round(a,4), round(b,4)]

# запустим функцию и запишем правильные ответы
ab_us = Kramer_method(x_us,y_us)
a_us = ab_us[0]
b_us = ab_us[1]
print ' 33[1m' + ' 33[4m' + "Оптимальные значения коэффициентов a и b:"  + ' 33[0m' 
print 'a =', a_us
print 'b =', b_us
print

# определим функцию для подсчета суммы квадратов ошибок
def errors_sq_Kramer_method(answers,x,y):
    list_errors_sq = []
    for i in range(len(x)):
        err = (answers[0] + answers[1]*x[i] - y[i])**2
        list_errors_sq.append(err)
    return sum(list_errors_sq)

# запустим функцию и запишем значение ошибки
error_sq = errors_sq_Kramer_method(ab_us,x_us,y_us)
print ' 33[1m' + ' 33[4m' + "Сумма квадратов отклонений" + ' 33[0m'
print error_sq
print

# замерим время расчета
# print ' 33[1m' + ' 33[4m' + "Время выполнения расчета суммы квадратов отклонений:" + ' 33[0m'
# % timeit error_sq = errors_sq_Kramer_method(ab,x_us,y_us)

Nakhu esinakho:

Ngakho-ke, amanani ama-coefficients atholiwe, isamba sokuchezuka okuyisikwele sesisunguliwe. Masidwebe umugqa oqondile ku-histogram ehlakazayo ngokuhambisana nama-coefficients atholiwe.

Ikhodi yomugqa wokuhlehla

# определим функцию для формирования массива рассчетных значений выручки
def sales_count(ab,x,y):
    line_answers = []
    [line_answers.append(ab[0]+ab[1]*x[i]) for i in range(len(x))]
    return line_answers

# построим графики
print 'Грфик№2 "Правильные и расчетные ответы"'
plt.plot(x_us,y_us,'o',color='green',markersize=16, label = '$True$ $answers$')
plt.plot(x_us, sales_count(ab_us,x_us,y_us), color='red',lw=4,
         label='$Function: a + bx,$ $where$ $a='+str(round(ab_us[0],2))+',$ $b='+str(round(ab_us[1],2))+'$')
plt.xlabel('$Months$', size=16)
plt.ylabel('$Sales$', size=16)
plt.legend(loc=1, prop={'size': 16})
plt.show()

Ishadi No. 2 “Izimpendulo ezifanele nezibaliwe”

Ungabheka igrafu yokuchezuka yenyanga ngayinye. Esimweni sethu, ngeke sithole inzuzo ebalulekile kuyo, kodwa sizokwanelisa ilukuluku lethu lokuthi isibalo sokuhlehla esilula somugqa sikubonisa kanjani ukuncika kwemali engenayo enyangeni yonyaka.

Ikhodi yeshadi lokuchezuka

# определим функцию для формирования массива отклонений в процентах
def error_per_month(ab,x,y):
    sales_c = sales_count(ab,x,y)
    errors_percent = []
    for i in range(len(x)):
        errors_percent.append(100*(sales_c[i]-y[i])/y[i])
    return errors_percent

# построим график
print 'График№3 "Отклонения по-месячно, %"'
plt.gca().bar(x_us, error_per_month(ab_us,x_us,y_us), color='brown')
plt.xlabel('Months', size=16)
plt.ylabel('Calculation error, %', size=16)
plt.show()

Ishadi No. 3 “Deviations, %”

Asiphelele, kodwa siwuqedile umsebenzi wethu.

Masibhale umsebenzi, ukuze sinqume ama-coefficients и isebenzisa umtapo wolwazi I-NumPy, ngokunembile, sizobhala imisebenzi emibili: eyodwa isebenzisa i-pseudoinverse matrix (ayinconyiwe ekusebenzeni, njengoba inqubo iyinkimbinkimbi futhi ayizinzile), enye isebenzisa i-matrix equation.

Ikhodi Yesixazululo Sokuhlaziya (NumPy)

# для начала добавим столбец с не изменяющимся значением в 1. 
# Данный столбец нужен для того, чтобы не обрабатывать отдельно коэффицент a
vector_1 = np.ones((x_np.shape[0],1))
x_np = table_zero[['x']].values # на всякий случай приведем в первичный формат вектор x_np
x_np = np.hstack((vector_1,x_np))

# проверим то, что все сделали правильно
print vector_1[0:3]
print x_np[0:3]
print '***************************************'
print

# напишем функцию, которая определяет значения коэффициентов a и b с использованием псевдообратной матрицы
def pseudoinverse_matrix(X, y):
    # задаем явный формат матрицы признаков
    X = np.matrix(X)
    # определяем транспонированную матрицу
    XT = X.T
    # определяем квадратную матрицу
    XTX = XT*X
    # определяем псевдообратную матрицу
    inv = np.linalg.pinv(XTX)
    # задаем явный формат матрицы ответов
    y = np.matrix(y)
    # находим вектор весов
    return (inv*XT)*y

# запустим функцию
ab_np = pseudoinverse_matrix(x_np, y_np)
print ab_np
print '***************************************'
print

# напишем функцию, которая использует для решения матричное уравнение
def matrix_equation(X,y):
    a = np.dot(X.T, X)
    b = np.dot(X.T, y)
    return np.linalg.solve(a, b)

# запустим функцию
ab_np = matrix_equation(x_np,y_np)
print ab_np

Ake siqhathanise isikhathi esichithwe ekunqumeni ama-coefficients и , ngokuhambisana nezindlela ezi-3 ezethulwe.

Ikhodi yokubala isikhathi sokubala

print ' 33[1m' + ' 33[4m' + "Время выполнения расчета коэффициентов без использования библиотеки NumPy:" + ' 33[0m'
% timeit ab_us = Kramer_method(x_us,y_us)
print '***************************************'
print
print ' 33[1m' + ' 33[4m' + "Время выполнения расчета коэффициентов с использованием псевдообратной матрицы:" + ' 33[0m'
%timeit ab_np = pseudoinverse_matrix(x_np, y_np)
print '***************************************'
print
print ' 33[1m' + ' 33[4m' + "Время выполнения расчета коэффициентов с использованием матричного уравнения:" + ' 33[0m'
%timeit ab_np = matrix_equation(x_np, y_np)

Ngenani elincane ledatha, umsebenzi "ozibhalayo" uphuma phambili, othola ama-coefficients usebenzisa indlela ye-Cramer.

Manje ungadlulela kwezinye izindlela zokuthola ama-coefficient и .

Ukwehla kweGradient

Okokuqala, ake sichaze ukuthi iyini i-gradient. Kalula nje, i-gradient iyisegimenti ebonisa isiqondiso sokukhula okuphezulu komsebenzi. Ngokufanisa nokuqwala intaba, lapho i-gradient ibheke khona kulapho umqansa ukhuphukela esiqongweni sentaba. Ukuthuthukisa isibonelo ngentaba, sikhumbula ukuthi empeleni sidinga ukwehla okuphakeme kakhulu ukuze sifinyelele endaweni ephansi ngokushesha ngangokunokwenzeka, okungukuthi, okungenani - indawo lapho umsebenzi ungakhuli noma unganciphi. Kuleli qophelo okuphuma kokunye kuzolingana noziro. Ngakho-ke, asidingi i-gradient, kodwa i-antigradient. Ukuze uthole i-antigradient udinga nje ukuphindaphinda i-gradient ngayo -1 (khipha okukodwa).

Ake sinake iqiniso lokuthi umsebenzi ungaba nama-minima amaningana, futhi sehlele kwenye yazo sisebenzisa i-algorithm ehlongozwayo ngezansi, ngeke sikwazi ukuthola obunye ubuncane, obungaba ngaphansi kwalowo otholakele. Ake siphumule, lokhu akulona usongo kithi! Esimweni sethu sibhekene nobuncane obuncane, kusukela emsebenzini wethu kugrafu kune-parabola evamile. Futhi njengoba sonke kufanele sazi kahle kakhulu esifundweni sethu sezibalo zesikole, i-parabola inenani eliphansi elilodwa kuphela.

Ngemva kokuthola ukuthi kungani sidinga i-gradient, nokuthi i-gradient iyisegimenti, okungukuthi, i-vector enezixhumanisi ezinikeziwe, ezingama-coefficient afanayo ncamashi. и singasebenzisa ukwehla kwe-gradient.

Ngaphambi kokuqala, ngiphakamisa ukuthi ufunde imisho embalwa nje mayelana ne-algorithm yokwehla:

• Sinquma ngendlela mbumbulu-okungahleliwe izixhumanisi zama-coefficients и . Esibonelweni sethu, sizonquma ama-coefficient aseduze kukaziro. Lona umkhuba ovamile, kodwa icala ngalinye lingase libe nomkhuba walo.
• Kusuka ku-coordinate khipha inani lokuphuma kokunye kokunye kwe-oda loku-1 endaweni . Ngakho-ke, uma i-derivative ilungile, khona-ke umsebenzi uyakhula. Ngakho-ke, ngokukhipha inani le-derivative, sizohamba ngendlela ehlukile yokukhula, okungukuthi, ekuqondeni kokwehla. Uma okuphuma kokunye kuyinegethivu, khona-ke umsebenzi kuleli phuzu uyancipha futhi ngokukhipha inani lokuphuma kokunye sihamba siye ohlangothini lokwehla.
• Senza umsebenzi ofanayo nge-coordinate : susa inani lokuphuma kokunye ingxenye endaweni .
• Ukuze ungagxumi phezu kobuncane bese undizela endaweni ejulile, kuyadingeka ukusetha usayizi wesinyathelo ngendlela yokwehla. Ngokuvamile, ungabhala i-athikili ephelele mayelana nendlela yokusetha isinyathelo ngendlela efanele nokuthi ungasishintsha kanjani phakathi nenqubo yokwehla ukuze unciphise izindleko zokubala. Kodwa manje sinomsebenzi ohluke kancane phambi kwethu, futhi sizosungula usayizi wesinyathelo sisebenzisa indlela yesayensi ye- "poke" noma, njengoba besho ngolimi oluvamile, ngokomthetho.
• Uma sesisuka ezixhumanisini ezinikeziwe и khipha amanani we-derivatives, sithola izixhumanisi ezintsha и . Sithatha isinyathelo esilandelayo (ukukhipha), kakade kusukela kuzixhumanisi ezibaliwe. Futhi ngakho-ke umjikelezo uqala ngokuphindaphindiwe, kuze kube yilapho kufinyelelwa ukuhlangana okudingekayo.

Konke! Manje sesilungele ukuyofuna umhosha ojulile we-Mariana Trench. Ake siqale.

Ikhodi yokwehla kwe-gradient

# напишем функцию градиентного спуска без использования библиотеки NumPy. 
# Функция на вход принимает диапазоны значений x,y, длину шага (по умолчанию=0,1), допустимую погрешность(tolerance)
def gradient_descent_usual(x_us,y_us,l=0.1,tolerance=0.000000000001):
    # сумма значений (все месяца)
    sx = sum(x_us)
    # сумма истинных ответов (выручка за весь период)
    sy = sum(y_us)
    # сумма произведения значений на истинные ответы
    list_xy = []
    [list_xy.append(x_us[i]*y_us[i]) for i in range(len(x_us))]
    sxy = sum(list_xy)
    # сумма квадратов значений
    list_x_sq = []
    [list_x_sq.append(x_us[i]**2) for i in range(len(x_us))]
    sx_sq = sum(list_x_sq)
    # количество значений
    num = len(x_us)
    # начальные значения коэффициентов, определенные псевдослучайным образом
    a = float(random.uniform(-0.5, 0.5))
    b = float(random.uniform(-0.5, 0.5))
    # создаем массив с ошибками, для старта используем значения 1 и 0
    # после завершения спуска стартовые значения удалим
    errors = [1,0]
    # запускаем цикл спуска
    # цикл работает до тех пор, пока отклонение последней ошибки суммы квадратов от предыдущей, не будет меньше tolerance
    while abs(errors[-1]-errors[-2]) > tolerance:
        a_step = a - l*(num*a + b*sx - sy)/num
        b_step = b - l*(a*sx + b*sx_sq - sxy)/num
        a = a_step
        b = b_step
        ab = [a,b]
        errors.append(errors_sq_Kramer_method(ab,x_us,y_us))
    return (ab),(errors[2:])

# запишем массив значений 
list_parametres_gradient_descence = gradient_descent_usual(x_us,y_us,l=0.1,tolerance=0.000000000001)


print ' 33[1m' + ' 33[4m' + "Значения коэффициентов a и b:" + ' 33[0m'
print 'a =', round(list_parametres_gradient_descence[0][0],3)
print 'b =', round(list_parametres_gradient_descence[0][1],3)
print


print ' 33[1m' + ' 33[4m' + "Сумма квадратов отклонений:" + ' 33[0m'
print round(list_parametres_gradient_descence[1][-1],3)
print



print ' 33[1m' + ' 33[4m' + "Количество итераций в градиентном спуске:" + ' 33[0m'
print len(list_parametres_gradient_descence[1])
print

Sangena phansi kakhulu ku-Mariana Trench futhi lapho sathola amanani e-coefficient afanayo и , okuyilokho kanye obekulindelwe.

Ake sibhukule futhi, kulokhu kuphela, imoto yethu yasolwandle izogcwala obunye ubuchwepheshe, okuwumtapo wolwazi. I-NumPy.

Ikhodi yokwehla kwegradient (NumPy)

# перед тем определить функцию для градиентного спуска с использованием библиотеки NumPy, 
# напишем функцию определения суммы квадратов отклонений также с использованием NumPy
def error_square_numpy(ab,x_np,y_np):
    y_pred = np.dot(x_np,ab)
    error = y_pred - y_np
    return sum((error)**2)

# напишем функцию градиентного спуска с использованием библиотеки NumPy. 
# Функция на вход принимает диапазоны значений x,y, длину шага (по умолчанию=0,1), допустимую погрешность(tolerance)
def gradient_descent_numpy(x_np,y_np,l=0.1,tolerance=0.000000000001):
    # сумма значений (все месяца)
    sx = float(sum(x_np[:,1]))
    # сумма истинных ответов (выручка за весь период)
    sy = float(sum(y_np))
    # сумма произведения значений на истинные ответы
    sxy = x_np*y_np
    sxy = float(sum(sxy[:,1]))
    # сумма квадратов значений
    sx_sq = float(sum(x_np[:,1]**2))
    # количество значений
    num = float(x_np.shape[0])
    # начальные значения коэффициентов, определенные псевдослучайным образом
    a = float(random.uniform(-0.5, 0.5))
    b = float(random.uniform(-0.5, 0.5))
    # создаем массив с ошибками, для старта используем значения 1 и 0
    # после завершения спуска стартовые значения удалим
    errors = [1,0]
    # запускаем цикл спуска
    # цикл работает до тех пор, пока отклонение последней ошибки суммы квадратов от предыдущей, не будет меньше tolerance
    while abs(errors[-1]-errors[-2]) > tolerance:
        a_step = a - l*(num*a + b*sx - sy)/num
        b_step = b - l*(a*sx + b*sx_sq - sxy)/num
        a = a_step
        b = b_step
        ab = np.array([[a],[b]])
        errors.append(error_square_numpy(ab,x_np,y_np))
    return (ab),(errors[2:])

# запишем массив значений 
list_parametres_gradient_descence = gradient_descent_numpy(x_np,y_np,l=0.1,tolerance=0.000000000001)

print ' 33[1m' + ' 33[4m' + "Значения коэффициентов a и b:" + ' 33[0m'
print 'a =', round(list_parametres_gradient_descence[0][0],3)
print 'b =', round(list_parametres_gradient_descence[0][1],3)
print


print ' 33[1m' + ' 33[4m' + "Сумма квадратов отклонений:" + ' 33[0m'
print round(list_parametres_gradient_descence[1][-1],3)
print

print ' 33[1m' + ' 33[4m' + "Количество итераций в градиентном спуске:" + ' 33[0m'
print len(list_parametres_gradient_descence[1])
print

Amanani e-coefficient и engaguquleki.

Ake sibheke ukuthi iphutha lishintshe kanjani ngesikhathi sokwehla kwegradient, okungukuthi, ukuthi isamba sokuchezuka okuyisikwele sishintshe kanjani ngesinyathelo ngasinye.

Ikhodi yokuhlela izibalo zokuchezuka okuyisikwele

print 'График№4 "Сумма квадратов отклонений по-шагово"'
plt.plot(range(len(list_parametres_gradient_descence[1])), list_parametres_gradient_descence[1], color='red', lw=3)
plt.xlabel('Steps (Iteration)', size=16)
plt.ylabel('Sum of squared deviations', size=16)
plt.show()

Igrafu No. 4 “Isamba sokuchezuka okuyisikwele ngesikhathi sokwehla kwegradient”

Kugrafu sibona ukuthi ngesinyathelo ngasinye iphutha liyancipha, futhi ngemva kwenani elithile lokuphindaphinda sibona umugqa ocishe uvundlile.

Okokugcina, ake silinganisele umehluko esikhathini sokwenza ikhodi:

Ikhodi yokunquma isikhathi sokubala sokwehla kwe-gradient

print ' 33[1m' + ' 33[4m' + "Время выполнения градиентного спуска без использования библиотеки NumPy:" + ' 33[0m'
%timeit list_parametres_gradient_descence = gradient_descent_usual(x_us,y_us,l=0.1,tolerance=0.000000000001)
print '***************************************'
print

print ' 33[1m' + ' 33[4m' + "Время выполнения градиентного спуска с использованием библиотеки NumPy:" + ' 33[0m'
%timeit list_parametres_gradient_descence = gradient_descent_numpy(x_np,y_np,l=0.1,tolerance=0.000000000001)

Mhlawumbe kukhona okungalungile esikwenzayo, kodwa futhi kuwumsebenzi olula “obhalwe ekhaya” ongawusebenzisi umtapo wolwazi. I-NumPy idlula isikhathi sokubala somsebenzi usebenzisa umtapo wolwazi I-NumPy.

Kodwa asimile, kodwa siphokophele ekufundeni enye indlela ethokozisayo yokuxazulula isibalo sokuhlehla esilula somugqa. Hlanganani!

Ukwehla kwe-Stochastic gradient

Ukuze uqonde ngokushesha umgomo wokusebenza kokwehla kwe-stochastic gradient, kungcono ukunquma umehluko wawo ekwehleni okujwayelekile kwe-gradient. Thina, endabeni yokwehla kwe-gradient, ezilinganisweni zokuphuma kokunye kwe и usebenzise izilinganiso zamanani azo zonke izici nezimpendulo eziyiqiniso ezitholakala kusampula (okungukuthi, isamba sakho konke и ). Ekwehleni kwe-stochastic gradient, ngeke sisebenzise wonke amanani akhona kusampula, kodwa esikhundleni salokho, khetha okungahleliwe lokho okubizwa ngokuthi inkomba yesampula futhi sisebenzise amanani ayo.

Isibonelo, uma inkomba inqunywa ukuthi iyinombolo 3 (ezintathu), sithatha amanani и , bese sifaka amanani esikhundleni sezibalo eziphuma kokunye futhi sinquma izixhumanisi ezintsha. Khona-ke, ngemva kokunquma izixhumanisi, siphinde sinqume ngokungahleliwe inkomba yesampula, sishintshe amanani ahambisana nenkomba kuma-equations ahlukene ngokwengxenye, futhi sinqume izixhumanisi ngendlela entsha. и njll. kuze kube yilapho ukuhlangana kuba luhlaza. Uma uthi nhlá, kungase kungabonakali sengathi lokhu kungasebenza nhlobo, kodwa kuyasebenza. Kuyiqiniso ukuthi kubalulekile ukuqaphela ukuthi iphutha alinciphi ngesinyathelo ngasinye, kodwa kukhona ukuthambekela ngokuqinisekile.

Yiziphi izinzuzo zokwehla kwe-stochastic gradient kunokwejwayelekile? Uma usayizi wethu wesampula mkhulu kakhulu futhi ukalwa ngamashumi ezinkulungwane zamanani, kulula kakhulu ukucubungula, ukusho, inkulungwane engahleliwe yawo, kunesampula yonke. Yilapho ukwehla kwe-stochastic gradient kuqala khona. Esimweni sethu, vele, ngeke siqaphele umehluko omkhulu.

Ake sibheke ikhodi.

Ikhodi yokwehla kwe-stochastic gradient

# определим функцию стох.град.шага
def stoch_grad_step_usual(vector_init, x_us, ind, y_us, l):
#     выбираем значение икс, которое соответствует случайному значению параметра ind 
# (см.ф-цию stoch_grad_descent_usual)
    x = x_us[ind]
#     рассчитывыаем значение y (выручку), которая соответствует выбранному значению x
    y_pred = vector_init[0] + vector_init[1]*x_us[ind]
#     вычисляем ошибку расчетной выручки относительно представленной в выборке
    error = y_pred - y_us[ind]
#     определяем первую координату градиента ab
    grad_a = error
#     определяем вторую координату ab
    grad_b = x_us[ind]*error
#     вычисляем новый вектор коэффициентов
    vector_new = [vector_init[0]-l*grad_a, vector_init[1]-l*grad_b]
    return vector_new


# определим функцию стох.град.спуска
def stoch_grad_descent_usual(x_us, y_us, l=0.1, steps = 800):
#     для самого начала работы функции зададим начальные значения коэффициентов
    vector_init = [float(random.uniform(-0.5, 0.5)), float(random.uniform(-0.5, 0.5))]
    errors = []
#     запустим цикл спуска
# цикл расчитан на определенное количество шагов (steps)
    for i in range(steps):
        ind = random.choice(range(len(x_us)))
        new_vector = stoch_grad_step_usual(vector_init, x_us, ind, y_us, l)
        vector_init = new_vector
        errors.append(errors_sq_Kramer_method(vector_init,x_us,y_us))
    return (vector_init),(errors)


# запишем массив значений 
list_parametres_stoch_gradient_descence = stoch_grad_descent_usual(x_us, y_us, l=0.1, steps = 800)

print ' 33[1m' + ' 33[4m' + "Значения коэффициентов a и b:" + ' 33[0m'
print 'a =', round(list_parametres_stoch_gradient_descence[0][0],3)
print 'b =', round(list_parametres_stoch_gradient_descence[0][1],3)
print


print ' 33[1m' + ' 33[4m' + "Сумма квадратов отклонений:" + ' 33[0m'
print round(list_parametres_stoch_gradient_descence[1][-1],3)
print

print ' 33[1m' + ' 33[4m' + "Количество итераций в стохастическом градиентном спуске:" + ' 33[0m'
print len(list_parametres_stoch_gradient_descence[1])

Sibheka ngokucophelela ama-coefficients futhi sizithole sibuza umbuzo "Kungenzeka kanjani lokhu?" Sithole amanye amanani we-coefficient и . Mhlawumbe ukwehla kwe-stochastic gradient kuthole amapharamitha angcono kakhulu esibalo? Ngeshwa cha. Kwanele ukubheka isamba sokuchezuka okuyisikwele futhi ubone ukuthi ngamavelu amasha wama-coefficients, iphutha likhulu. Asijahile ukulahla ithemba. Ake sakhe igrafu yoshintsho lwephutha.

Ikhodi yokuhlela isamba sokuchezuka okusikwele ekwehleni kwegradient ye-stochastic

print 'График №5 "Сумма квадратов отклонений по-шагово"'
plt.plot(range(len(list_parametres_stoch_gradient_descence[1])), list_parametres_stoch_gradient_descence[1], color='red', lw=2)
plt.xlabel('Steps (Iteration)', size=16)
plt.ylabel('Sum of squared deviations', size=16)
plt.show()

Igrafu No. 5 “Isamba sokuchezuka okusikwele ngesikhathi sokwehla kwe-stochastic gradient”

Uma sibheka isheduli, yonke into ihamba kahle futhi manje sizolungisa yonke into.

Pho kwenzekani? Lokhu okulandelayo kwenzeka. Uma sikhetha inyanga ngokungahleliwe, kuba okwenyanga ekhethiwe lapho i-algorithm yethu ifuna ukunciphisa iphutha ekubaleni imali engenayo. Bese sikhetha enye inyanga bese siphinda ukubala, kodwa sinciphisa iphutha lenyanga yesibili ekhethiwe. Manje khumbula ukuthi izinyanga ezimbili zokuqala zichezuka kakhulu emugqeni wezibalo zokuhlehla ezilula zomugqa. Lokhu kusho ukuthi uma noma iyiphi yalezi zinyanga ezimbili ikhethiwe, ngokunciphisa iphutha lazo zonke, i-algorithm yethu ikhulisa kakhulu iphutha layo yonke isampula. Pho yini okumele uyenze? Impendulo ilula: udinga ukunciphisa isinyathelo sokwehla. Phela, ngokunciphisa isinyathelo sokwehla, iphutha lizoyeka "ukweqa" phezulu naphansi. Noma kunalokho, iphutha "lokweqa" ngeke liyeke, kodwa ngeke likwenze ngokushesha :) Ake sihlole.

Ikhodi yokusebenzisa i-SGD ngezinyuso ezincane

# запустим функцию, уменьшив шаг в 100 раз и увеличив количество шагов соответсвующе 
list_parametres_stoch_gradient_descence = stoch_grad_descent_usual(x_us, y_us, l=0.001, steps = 80000)

print ' 33[1m' + ' 33[4m' + "Значения коэффициентов a и b:" + ' 33[0m'
print 'a =', round(list_parametres_stoch_gradient_descence[0][0],3)
print 'b =', round(list_parametres_stoch_gradient_descence[0][1],3)
print


print ' 33[1m' + ' 33[4m' + "Сумма квадратов отклонений:" + ' 33[0m'
print round(list_parametres_stoch_gradient_descence[1][-1],3)
print



print ' 33[1m' + ' 33[4m' + "Количество итераций в стохастическом градиентном спуске:" + ' 33[0m'
print len(list_parametres_stoch_gradient_descence[1])

print 'График №6 "Сумма квадратов отклонений по-шагово"'
plt.plot(range(len(list_parametres_stoch_gradient_descence[1])), list_parametres_stoch_gradient_descence[1], color='red', lw=2)
plt.xlabel('Steps (Iteration)', size=16)
plt.ylabel('Sum of squared deviations', size=16)
plt.show()

Igrafu No. 6 “Isamba sokuchezuka okuyisikwele ngesikhathi sokwehla kwe-stochastic gradient (izinyathelo eziyizinkulungwane ezingu-80)”

Ama-coefficients athuthukile, kodwa namanje awalungile. Ngokuqagela, lokhu kungalungiswa ngale ndlela. Sikhetha, isibonelo, ekuphindaphindweni okungu-1000 kokugcina amanani ama-coefficients okwenziwa ngawo iphutha elincane. Yiqiniso, kulokhu kuzodingeka futhi sibhale phansi amanani ama-coefficients ngokwawo. Ngeke sikwenze lokhu, kodwa kunalokho sinake isimiso. Kubukeka bushelelezi futhi iphutha libonakala lehla ngokulinganayo. Empeleni lokhu akulona iqiniso. Ake sibheke ukuphindaphinda okungu-1000 kokuqala futhi sikuqhathanise nokugcina.

Ikhodi yeshadi le-SGD (izinyathelo zokuqala eziyi-1000)

print 'График №7 "Сумма квадратов отклонений по-шагово. Первые 1000 итераций"'
plt.plot(range(len(list_parametres_stoch_gradient_descence[1][:1000])), 
         list_parametres_stoch_gradient_descence[1][:1000], color='red', lw=2)
plt.xlabel('Steps (Iteration)', size=16)
plt.ylabel('Sum of squared deviations', size=16)
plt.show()

print 'График №7 "Сумма квадратов отклонений по-шагово. Последние 1000 итераций"'
plt.plot(range(len(list_parametres_stoch_gradient_descence[1][-1000:])), 
         list_parametres_stoch_gradient_descence[1][-1000:], color='red', lw=2)
plt.xlabel('Steps (Iteration)', size=16)
plt.ylabel('Sum of squared deviations', size=16)
plt.show()

Igrafu No. 7 “Isamba sokuchezuka kwesikwele SGD (izinyathelo zokuqala ezingu-1000)”

Igrafu No. 8 “Isamba sokuchezuka okuyisikwele SGD (izinyathelo zokugcina ezingu-1000)”

Ekuqaleni kokwehla, sibona iphutha elilinganayo kanye nokwehla okukhulu. Ekuphindaphindweni kokugcina, sibona ukuthi iphutha lizungeza futhi lizungeze inani le-1,475 futhi ngezinye izikhathi lilingana naleli nani eliphelele, kodwa-ke liyakhuphuka... Ngiyaphinda, ungabhala phansi amanani we- ama-coefficients и , bese ukhetha lezo iphutha elincane kuzo. Kodwa-ke, sibe nenkinga enkulu kakhulu: bekufanele sithathe izinyathelo eziyizinkulungwane ezingama-80 (bona ikhodi) ukuze sithole amanani asondele kokulungile. Futhi lokhu vele kuyaphikisana nomqondo wokonga isikhathi sokubala ngokwehla kwe-stochastic gradient okuhlobene nokwehla kwe-gradient. Yini engalungiswa futhi ithuthukiswe? Akunzima ukuqaphela ukuthi ekuphindaphindweni kokuqala siyehla ngokuzethemba futhi, ngakho-ke, kufanele sishiye isinyathelo esikhulu ekuphindaphindweni kokuqala futhi sinciphise isinyathelo njengoba siqhubekela phambili. Ngeke sikwenze lokhu kulesi sihloko - sesivele side kakhulu. Labo abafisayo bangazicabangela bona ukuthi bangakwenza kanjani lokhu, akunzima :)

Manje masenze ukwehla kwe-stochastic gradient sisebenzisa umtapo wolwazi I-NumPy (futhi masingakhubeki ematsheni esiwahlonze ekuqaleni)

Ikhodi yokwehla kwe-Stochastic Gradient (NumPy)

# для начала напишем функцию градиентного шага
def stoch_grad_step_numpy(vector_init, X, ind, y, l):
    x = X[ind]
    y_pred = np.dot(x,vector_init)
    err = y_pred - y[ind]
    grad_a = err
    grad_b = x[1]*err
    return vector_init - l*np.array([grad_a, grad_b])

# определим функцию стохастического градиентного спуска
def stoch_grad_descent_numpy(X, y, l=0.1, steps = 800):
    vector_init = np.array([[np.random.randint(X.shape[0])], [np.random.randint(X.shape[0])]])
    errors = []
    for i in range(steps):
        ind = np.random.randint(X.shape[0])
        new_vector = stoch_grad_step_numpy(vector_init, X, ind, y, l)
        vector_init = new_vector
        errors.append(error_square_numpy(vector_init,X,y))
    return (vector_init), (errors)

# запишем массив значений 
list_parametres_stoch_gradient_descence = stoch_grad_descent_numpy(x_np, y_np, l=0.001, steps = 80000)

print ' 33[1m' + ' 33[4m' + "Значения коэффициентов a и b:" + ' 33[0m'
print 'a =', round(list_parametres_stoch_gradient_descence[0][0],3)
print 'b =', round(list_parametres_stoch_gradient_descence[0][1],3)
print


print ' 33[1m' + ' 33[4m' + "Сумма квадратов отклонений:" + ' 33[0m'
print round(list_parametres_stoch_gradient_descence[1][-1],3)
print



print ' 33[1m' + ' 33[4m' + "Количество итераций в стохастическом градиентном спуске:" + ' 33[0m'
print len(list_parametres_stoch_gradient_descence[1])
print

Amanani acishe afana nalapho ehla ngaphandle kokusebenzisa I-NumPy. Nokho, lokhu kunengqondo.

Ake sithole ukuthi ukwehla kwe-stochastic gradient kusithatha isikhathi esingakanani.

Ikhodi yokunquma isikhathi sokubala se-SGD (izinyathelo eziyizinkulungwane ezingama-80)

print ' 33[1m' + ' 33[4m' +
"Время выполнения стохастического градиентного спуска без использования библиотеки NumPy:"
+ ' 33[0m'
%timeit list_parametres_stoch_gradient_descence = stoch_grad_descent_usual(x_us, y_us, l=0.001, steps = 80000)
print '***************************************'
print

print ' 33[1m' + ' 33[4m' +
"Время выполнения стохастического градиентного спуска с использованием библиотеки NumPy:"
+ ' 33[0m'
%timeit list_parametres_stoch_gradient_descence = stoch_grad_descent_numpy(x_np, y_np, l=0.001, steps = 80000)

Ukuqhubekela phambili ehlathini, amafu amnyama: futhi, ifomula "ezibhalile" ibonisa umphumela omuhle kakhulu. Konke lokhu kusikisela ukuthi kufanele kube nezindlela ezicashile zokusebenzisa umtapo wolwazi I-NumPy, okusheshisa ngempela ukusebenza kokubala. Kulesi sihloko ngeke sifunde ngabo. Kuzoba nokuthile ongacabanga ngakho ngesikhathi sakho esikhululekile :)

Sifingqa

Ngaphambi kokufingqa, ngithanda ukuphendula umbuzo okungenzeka uvele kumfundi wethu othandekayo. Kungani, empeleni, "ukuhlukunyezwa" okunjalo nokwehla, kungani sidinga ukuhamba sehla senyuka intaba (ikakhulukazi phansi) ukuze sithole indawo ephansi eyigugu, uma ezandleni zethu siphethe idivayisi enamandla futhi elula, uhlobo lwesixazululo sokuhlaziya, esisithumela ngokushesha endaweni elungile?

Impendulo yalo mbuzo itholakala phezulu. Manje sibheke isibonelo esilula kakhulu, lapho impendulo yeqiniso ikhona kuncike kuphawu olulodwa . Awukuboni lokhu kaningi empilweni, ngakho-ke ake sicabange ukuthi sinezimpawu ezingu-2, 30, 50 noma ngaphezulu. Ake sengeze kulokhu izinkulungwane, noma amashumi ezinkulungwane zamavelu kusibaluli ngasinye. Kulesi simo, isisombululo sokuhlaziya singase singamelani nokuhlolwa futhi sehluleke. Ngokulandelayo, ukwehla kwe-gradient nokuhluka kwakho kuzosisondeza kancane kancane kumgomo - ubuncane bomsebenzi. Futhi ungakhathazeki ngesivinini - cishe sizobheka izindlela ezizosivumela ukuthi sibeke futhi silawule ubude besinyathelo (okungukuthi, isivinini).

Futhi manje isifinyezo sangempela esifushane.

Okokuqala, ngithemba ukuthi izinto ezivezwe esihlokweni zizosiza ekuqaleni “ososayensi bedatha” ekuqondeni indlela yokuxazulula izilinganiso zokuhlehla zomugqa ezilula (hhayi nje kuphela).

Okwesibili, sibheke izindlela ezimbalwa zokuxazulula i-equation. Manje, kuye ngesimo, singakhetha lowo ofaneleka kakhulu ukuxazulula inkinga.

Okwesithathu, sibone amandla ezilungiselelo ezengeziwe, okungukuthi ubude besinyathelo sokwehla kwe-gradient. Le pharamitha ayikwazi ukunganakwa. Njengoba kuphawuliwe ngenhla, ukuze kuncishiswe izindleko zokubala, ubude besinyathelo kufanele bushintshwe ngesikhathi sokwehla.

Okwesine, esimweni sethu, imisebenzi "ebhalwe ekhaya" ibonise imiphumela yesikhathi engcono kakhulu yokubala. Lokhu mhlawumbe kungenxa yokungasebenzisi kahle kakhulu kwamakhono omtapo wolwazi I-NumPy. Kodwa noma ngabe kunjalo, isiphetho esilandelayo siyazisikisela. Ngakolunye uhlangothi, ngezinye izikhathi kufanelekile ukubuza imibono emisiwe, futhi ngakolunye uhlangothi, akufanelekile ngaso sonke isikhathi ukuhlanganisa yonke into - ngokuphambene nalokho, ngezinye izikhathi indlela elula yokuxazulula inkinga iphumelela kakhulu. Futhi njengoba inhloso yethu bekuwukuhlaziya izindlela ezintathu zokuxazulula isibalo esilula sokuhlehla komugqa, ukusetshenziswa kwemisebenzi "ezibhale ngokwakho" bekwanele kithi.

Imibhalo (noma into efana naleyo)

1. Ukwehla komugqa

http://statistica.ru/theory/osnovy-lineynoy-regressii/

2. Indlela encane yezikwele

mathprofi.ru/metod_naimenshih_kvadratov.html

3. Okuphuma kokunye

www.mathprofi.ru/chastnye_proizvodnye_primery.html

I-4. I-Gradient

mathprofi.ru/proizvodnaja_po_napravleniju_i_gradient.html

5. Ukwehla kwe-gradient

habr.com/zu/post/471458

habr.com/zu/post/307312

artemarakcheev.com//2017-12-31/linear_regression

6. Umtapo wolwazi we-NumPy

docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.linalg.solve.html

docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.linalg.pinv.html

pythonworld.ru/numpy/2.html

Source: www.habr.com