{"id":56186,"date":"2020-02-06T00:00:00","date_gmt":"2020-02-05T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/povtornaya-obrabotka-sobytij-poluchennyh-iz-kafka"},"modified":"2020-02-18T14:04:24","modified_gmt":"2020-02-18T11:04:24","slug":"povtornaya-obrabotka-sobytij-poluchennyh-iz-kafka","status":"publish","type":"post","link":"https:\/\/prohoster.info\/az\/blog\/administrirovanie\/povtornaya-obrabotka-sobytij-poluchennyh-iz-kafka","title":{"rendered":"Kafka-dan al\u0131nan hadis\u0259l\u0259rin t\u0259krar emal\u0131","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Kafka-dan<\/p>\n

<\/p>\n

Salam, Habr.<\/p>\n

<\/p>\n

Son zamanlarda t\u0259cr\u00fcb\u0259mi b\u00f6l\u00fc\u015fd\u00fcm<\/a><\/noindex> Kafka Producer v\u0259 Consumer \u00fc\u00e7\u00fcn komandam\u0131z\u0131n daha \u00e7ox istifad\u0259 etdiyi parametrl\u0259r bar\u0259d\u0259. Bu m\u0259qal\u0259d\u0259, xarici sistemin m\u00fcv\u0259qq\u0259ti \u0259l\u00e7atan olmamas\u0131 n\u0259tic\u0259sind\u0259 Kafka-dan al\u0131nan hadis\u0259nin t\u0259krar emal\u0131n\u0131 nec\u0259 t\u0259\u015fkil etdiyimizi anlatmaq ist\u0259yir\u0259m.<\/p>\n

<\/p>\n

M\u00fcasir t\u0259tbiql\u0259r \u00e7ox m\u00fcr\u0259kk\u0259b bir m\u00fchitd\u0259 f\u0259aliyy\u0259t g\u00f6st\u0259rir. M\u00fcasir texnoloji y\u0131\u011f\u0131nla sar\u0131lan biznes m\u0259nti\u011fi, Kubernetes v\u0259 ya OpenShift kimi orkestrator t\u0259r\u0259find\u0259n idar\u0259 olunan Docker imici i\u00e7ind\u0259dir v\u0259 dig\u0259r t\u0259tbiql\u0259rl\u0259 v\u0259 ya enterprise h\u0259ll\u0259ri il\u0259 fiziki v\u0259 virtual y\u00f6nl\u0259ndiricil\u0259r silsil\u0259si vasit\u0259sil\u0259 \u00fcnsiyy\u0259t qurur. Bel\u0259 bir m\u00fchitd\u0259 h\u0259r zaman bir \u015feyin pozulmas\u0131 m\u00fcmk\u00fcnd\u00fcr, buna g\u00f6r\u0259 d\u0259 xarici sisteml\u0259rd\u0259n birinin \u0259l\u00e7atan olmamas\u0131 hal\u0131nda hadis\u0259l\u0259rin t\u0259krar emal\u0131 bizim biznes prosesl\u0259rimizin m\u00fch\u00fcm bir hiss\u0259sidir.<\/p>\n

<\/a><\/noindex><\/p>\n

Kafka-dan \u0259vv\u0259l<\/h2>\n

<\/p>\n

Layih\u0259d\u0259 \u0259vv\u0259lc\u0259 asinxron mesaj g\u00f6nd\u0259rm\u0259 \u00fc\u00e7\u00fcn IBM MQ istifad\u0259 edirdik. Xidm\u0259tin i\u015f prosesi zaman\u0131 h\u0259r hans\u0131 bir x\u0259ta ba\u015f verdikd\u0259, al\u0131nan mesaj dead-letter-queue (DLQ) daxilind\u0259 yar\u0131d\u0131l\u0131b manual t\u0259hlil \u00fc\u00e7\u00fcn g\u00f6nd\u0259ril\u0259 bil\u0259rdi. DLQ, giri\u015f qov\u015fa\u011f\u0131n\u0131n yan\u0131nda yarad\u0131l\u0131rd\u0131 v\u0259 mesaj\u0131n k\u00f6\u00e7\u00fcr\u00fclm\u0259si IBM MQ daxilind\u0259 ba\u015f verirdi. <\/p>\n

<\/p>\n

\u018fg\u0259r x\u0259ta m\u00fcv\u0259qq\u0259ti xarakter da\u015f\u0131y\u0131rd\u0131sa v\u0259 biz bunu m\u00fc\u0259yy\u0259n ed\u0259 bilirdiks\u0259 (m\u0259s\u0259l\u0259n, HTTP \u00e7a\u011f\u0131r\u0131\u015f\u0131 zaman\u0131 ResourceAccessException v\u0259 ya MongoDb-y\u0259 edil\u0259n sor\u011fu zaman\u0131 MongoTimeoutException), o zaman t\u0259krar \u00e7a\u011f\u0131rmalar strategiyas\u0131 devreye girirdi. T\u0259tbiqin m\u0259ntiqinin \u015fax\u0259l\u0259nm\u0259sind\u0259n as\u0131l\u0131 olmayaraq, ilkin mesaj ya gecikdirilmi\u015f g\u00f6nd\u0259ri\u015f \u00fc\u00e7\u00fcn sistem s\u0131ra\u0131na, ya da bir zamanlar mesajlar\u0131 t\u0259krar g\u00f6nd\u0259rm\u0259k \u00fc\u00e7\u00fcn haz\u0131rlanm\u0131\u015f ayr\u0131 bir t\u0259tbiq\u0259 k\u00f6\u00e7\u00fcr\u00fcl\u00fcrd\u00fc. Bu zaman mesaj\u0131n ba\u015fl\u0131\u011f\u0131na g\u00f6nd\u0259ri\u015f say\u0131 yaz\u0131l\u0131rd\u0131 ki, bu da gecikm\u0259 interval\u0131 v\u0259 ya t\u0259tbiq s\u0259viyy\u0259sind\u0259 strategiyan\u0131n sona \u00e7atmas\u0131na ba\u011fl\u0131d\u0131r. Strategiyan\u0131n sonuna \u00e7atm\u0131\u015f\u0131qsa, lakin xarici sistem h\u0259l\u0259 d\u0259 \u0259l\u00e7atan deyils\u0259, mesaj DLQ-ya manual t\u0259hlil \u00fc\u00e7\u00fcn g\u00f6nd\u0259ril\u0259c\u0259k.<\/p>\n

<\/p>\n

H\u0259ll axtar\u0131\u015f\u0131<\/h2>\n

<\/p>\n

\u0130nternetd\u0259 axtar\u0131\u015f etdikc\u0259<\/a><\/noindex>, a\u015fa\u011f\u0131dak\u0131lar\u0131 tapa bil\u0259rsiniz h\u0259ll<\/a><\/noindex>. Q\u0131sa des\u0259k, h\u0259r gecikm\u0259 intervallar\u0131na bir m\u00f6vzu a\u00e7maq v\u0259 t\u0259tbiq t\u0259r\u0259find\u0259 m\u00fcvafiq gecikm\u0259 il\u0259 mesajlar\u0131 oxuyan Consumer-l\u0259ri h\u0259yata ke\u00e7irm\u0259k t\u0259klif olunur. <\/p>\n

<\/p>\n

\"Kafka-dan<\/p>\n

<\/p>\n

Bax\u0131\u015flar\u0131n \u00e7oxsayl\u0131 m\u00fcsb\u0259t olmas\u0131na baxmayaraq, bu, m\u0259n\u0259 tamamil\u0259 u\u011furlu g\u00f6r\u00fcnm\u00fcr. Birincisi, ona g\u00f6r\u0259 ki, inki\u015faf etdiricinin biznes t\u0259l\u0259bl\u0259rini yerin\u0259 yetirm\u0259kd\u0259n \u0259lav\u0259, t\u0259svir olunan mexanizmi h\u0259yata ke\u00e7irm\u0259k \u00fc\u00e7\u00fcn \u00e7ox vaxt s\u0259rf etm\u0259si laz\u0131m olacaq.<\/p>\n

<\/p>\n

Bundan \u0259lav\u0259, \u0259g\u0259r Kafka klasterind\u0259 giri\u015f idar\u0259etm\u0259si aktivdirs\u0259, topikl\u0259rin yarad\u0131lmas\u0131 v\u0259 onlara laz\u0131m olan giri\u015fl\u0259rin t\u0259min edilm\u0259si \u00fc\u00e7\u00fcn bir q\u0259d\u0259r vaxt s\u0259rf olunmal\u0131d\u0131r. Bununla yana\u015f\u0131, h\u0259r bir yenid\u0259n c\u0259hd olunan topik \u00fc\u00e7\u00fcn retention.ms parametrini d\u00fczg\u00fcn se\u00e7m\u0259k laz\u0131md\u0131r ki, mesajlar t\u0259krar g\u00f6nd\u0259rilsin v\u0259 itirilm\u0259sin. Giri\u015fl\u0259ri h\u0259yata ke\u00e7irm\u0259k v\u0259 sor\u011funu h\u0259r bir m\u00f6vcud v\u0259 ya yeni xidm\u0259t \u00fc\u00e7\u00fcn t\u0259krarlamaq laz\u0131m olacaq.<\/p>\n

<\/p>\n

G\u0259lin indi spring-in \u00fcmumilikd\u0259 v\u0259 spring-kafka-n\u0131n x\u00fcsusil\u0259 mesajlar\u0131n t\u0259krar emal\u0131 \u00fc\u00e7\u00fcn t\u0259qdim etdiyi mexanizml\u0259r\u0259 baxaq. Spring-kafka, m\u00fcxt\u0259lif BackOffPolicy-l\u0259ri idar\u0259 etm\u0259k \u00fc\u00e7\u00fcn abstractions t\u0259qdim ed\u0259n spring-retry-\u0259 as\u0131l\u0131l\u0131q yarad\u0131r. Bu, olduqca \u00e7evik bir al\u0259tdir, lakin onun \u0259h\u0259miyy\u0259tli bir \u00e7at\u0131\u015fmazl\u0131\u011f\u0131, t\u0259krar g\u00f6nd\u0259rilm\u0259si \u00fc\u00e7\u00fcn mesajlar\u0131n t\u0259tbiqin yadda\u015f\u0131nda saxlan\u0131lmas\u0131d\u0131r. Bu, t\u0259tbiqin yenil\u0259m\u0259si v\u0259 ya istismar zaman\u0131 bir s\u0259hv s\u0259b\u0259bind\u0259n yenid\u0259n ba\u015flamas\u0131 n\u0259tic\u0259sind\u0259 b\u00fct\u00fcn g\u00f6zl\u0259m\u0259d\u0259 olan mesajlar\u0131n itirilm\u0259sin\u0259 s\u0259b\u0259b olur. Bu m\u0259s\u0259l\u0259 bizim sistemimiz \u00fc\u00e7\u00fcn kritik oldu\u011fundan, biz onu daha da ara\u015fd\u0131rmad\u0131q.<\/p>\n

<\/p>\n

K\u00f6m\u0259k\u00e7i spring-kafka bir ne\u00e7\u0259 ContainerAwareErrorHandler implementasiyas\u0131 t\u0259qdim edir, m\u0259s\u0259l\u0259n SeekToCurrentErrorHandler<\/a><\/noindex>, bununla, bir s\u0259hv yaranarsa offset-m\u0259 d\u0259yi\u015fm\u0259d\u0259n mesaj\u0131 sonra emal etm\u0259k m\u00fcmk\u00fcnd\u00fcr. Spring-kafka 2.3 versiyas\u0131ndan ba\u015flayaraq BackOffPolicy t\u0259yin etm\u0259k imkan\u0131 t\u0259min edilib.<\/p>\n

<\/p>\n

Bu yana\u015fma, t\u0259krar emal olunan mesajlar\u0131n t\u0259tbiqin yenid\u0259n ba\u015flamas\u0131n\u0131 ya\u015famas\u0131na imkan tan\u0131y\u0131r, lakin DLQ mexanizmi h\u0259l\u0259 d\u0259 m\u00f6vcud deyil. Biz 2019-cu ilin \u0259vv\u0259lind\u0259 bu variant\u0131 se\u00e7dik, optimistc\u0259sin\u0259 DLQ-ya ehtiyac olmayaca\u011f\u0131n\u0131 d\u00fc\u015f\u00fcn\u0259r\u0259k (\u015fans\u0131m\u0131z g\u0259tirdi v\u0259 ger\u00e7\u0259kd\u0259n d\u0259 bir ne\u00e7\u0259 ayl\u0131q t\u0259tbiq istismar\u0131 \u0259rzind\u0259 DLQ-ya ehtiyac olmad\u0131). M\u00fcv\u0259qq\u0259ti s\u0259hvl\u0259r SeekToCurrentErrorHandler-in i\u015f\u0259 d\u00fc\u015fm\u0259sin\u0259 s\u0259b\u0259b oldu. Dig\u0259r s\u0259hvl\u0259r is\u0259 loga yaz\u0131l\u0131r, offset-in d\u0259yi\u015fm\u0259sin\u0259 yol a\u00e7\u0131r v\u0259 emal n\u00f6vb\u0259ti mesajla davam edir.<\/p>\n

<\/p>\n

N\u0259tic\u0259 q\u0259rar\u0131<\/h2>\n

<\/p>\n

SeekToCurrentErrorHandler-a \u0259saslanan t\u0259tbiqetm\u0259nin h\u0259yata ke\u00e7irilm\u0259si, biz\u0259 \u00f6z mesajlar\u0131n yenid\u0259n g\u00f6nd\u0259rilm\u0259si mexanizmini inki\u015faf etdirm\u0259k \u00fc\u00e7\u00fcn ilham verdi.<\/p>\n

<\/p>\n

\u018fvv\u0259lc\u0259 biz m\u00f6vcud t\u0259cr\u00fcb\u0259ni istifad\u0259 etm\u0259k v\u0259 onu t\u0259tbiqin m\u0259ntiqin\u0259 g\u00f6r\u0259 geni\u015fl\u0259ndirm\u0259k ist\u0259yirdik. X\u0259tti m\u0259ntiq\u0259 malik olan t\u0259tbiq \u00fc\u00e7\u00fcn yeni mesajlar\u0131n oxunmas\u0131n\u0131n dayand\u0131r\u0131lmas\u0131 \u00fc\u00e7\u00fcn ki\u00e7ik bir m\u00fcdd\u0259t t\u0259yin etm\u0259k optimal olard\u0131, bu m\u00fcdd\u0259t is\u0259 t\u0259krar \u00e7a\u011f\u0131rma strategiyas\u0131 \u00e7\u0259r\u00e7iv\u0259sind\u0259 m\u00fc\u0259yy\u0259n edilir. Dig\u0259r t\u0259tbiql\u0259r \u00fc\u00e7\u00fcn is\u0259 weblayih\u0259y\u0259 t\u0259min etm\u0259si \u00fc\u00e7\u00fcn t\u0259k bir n\u00f6qt\u0259nin olmas\u0131 arzuolunand\u0131r ki, bu, iki yana\u015fma \u00fc\u00e7\u00fcn t\u0259krar \u00e7a\u011f\u0131rma strategiyas\u0131n\u0131 h\u0259yata ke\u00e7irir. Bununla yana\u015f\u0131, bu t\u0259k n\u00f6qt\u0259 h\u0259m d\u0259 h\u0259r iki yana\u015fma \u00fc\u00e7\u00fcn DLQ funksionall\u0131\u011f\u0131na sahib olmal\u0131d\u0131r.<\/p>\n

<\/p>\n

T\u0259krar \u00e7a\u011f\u0131rma strategiyas\u0131, m\u00fcv\u0259qq\u0259ti x\u0259tan\u0131n ba\u015f verm\u0259si zaman\u0131 n\u00f6vb\u0259ti intervallar\u0131 almaqdan m\u0259sul olan t\u0259tbiqin i\u00e7ind\u0259 saxlanmal\u0131d\u0131r.<\/p>\n

<\/p>\n

X\u0259tti m\u0259ntiq\u0259 malik t\u0259tbiq \u00fc\u00e7\u00fcn Consumer-in dayanmas\u0131<\/h3>\n

<\/p>\n

spring-kafka il\u0259 i\u015fl\u0259dikd\u0259, Consumer-in dayanma kodu t\u0259xmin\u0259n bel\u0259 g\u00f6r\u00fcn\u0259 bil\u0259r:<\/p>\n

<\/p>\n

public void pauseListenerContainer(MessageListenerContainer listenerContainer, \n                                   Instant retryAt) {\n        if (nonNull(retryAt) && listenerContainer.isRunning()) {\n            listenerContainer.stop();\n            taskScheduler.schedule(() -> listenerContainer.start(), retryAt);\n            return;\n        }\n        \/\/ DLQ-ya g\u00f6nd\u0259r<\/code><\/pre>\n
<\/p>\n
M\u00fcxt\u0259lif retryAt - MessageListenerContainer-in yenid\u0259n i\u015f\u0259 sal\u0131nmas\u0131 \u00fc\u00e7\u00fcn vaxtd\u0131r, \u0259g\u0259r o, h\u0259l\u0259 d\u0259 i\u015fl\u0259yirs\u0259. Yenid\u0259n ba\u015flamas\u0131 TaskScheduler-d\u0259 ger\u00e7ekle\u015ftirilen ayr\u0131 bir ipd\u0259 ba\u015f ver\u0259c\u0259k, onun implementasiyas\u0131n\u0131 da spring t\u0259klif edir. <\/p>\n
<\/p>\n
retryAt d\u0259y\u0259rini a\u015fa\u011f\u0131dak\u0131 kimi tap\u0131r\u0131q:<\/p>\n
<\/p>\n
    \n
  1. Yenid\u0259n \u00e7a\u011f\u0131rmalar\u0131n say\u011fac\u0131n\u0131n d\u0259y\u0259ri axtar\u0131l\u0131r.<\/li>\n
  2. Say\u011fac\u0131n d\u0259y\u0259rin\u0259 uy\u011fun olaraq, t\u0259krar \u00e7a\u011f\u0131rma strategiyas\u0131nda cari gecikm\u0259 intervallar\u0131 axtar\u0131l\u0131r. Strategiyam\u0131z t\u0259tbiqin i\u00e7ind\u0259 elan edilir, onun saxlanmas\u0131 \u00fc\u00e7\u00fcn JSON format\u0131n\u0131 se\u00e7dik.<\/li>\n
  3. JSON massivind\u0259 tap\u0131lan intervallar, emal\u0131n t\u0259krar\u0131 \u00fc\u00e7\u00fcn laz\u0131m olan saniy\u0259l\u0259rin say\u0131n\u0131 ehtiva edir. Bu say\u0131 cari vaxta \u0259lav\u0259 olunur, n\u0259tic\u0259d\u0259 retryAt \u00fc\u00e7\u00fcn d\u0259y\u0259r meydana g\u0259lir.<\/li>\n
  4. \u018fg\u0259r interval tap\u0131lmasa, onda retryAt d\u0259y\u0259ri null olur v\u0259 mesaj DLQ-ya g\u00f6nd\u0259rilir ki, bu da \u0259l il\u0259 analiz edil\u0259c\u0259k.<\/li>\n<\/ol>\n
    <\/p>\n
    Bu yana\u015fmada, indiki zaman \u00fc\u00e7\u00fcn emal olunan h\u0259r mesaj \u00fc\u00e7\u00fcn h\u0259r d\u0259f\u0259 \u00e7a\u011f\u0131rmalar\u0131n say\u0131n\u0131 saxlamaq laz\u0131md\u0131r, m\u0259s\u0259l\u0259n, t\u0259tbiqin yadda\u015f\u0131nda. C\u0259hd say\u0131n\u0131 yadda\u015fda saxlamaq bu yana\u015fma \u00fc\u00e7\u00fcn kritik deyil, \u00e7\u00fcnki x\u0259tti m\u0259ntiq\u0259 malik bir t\u0259tbiq \u00fcmumilikd\u0259 emal\u0131 h\u0259yata ke\u00e7ir\u0259 bilmir. Spring-retry il\u0259 f\u0259rqli olaraq, t\u0259tbiqin yenid\u0259n ba\u015flad\u0131lmas\u0131, b\u00fct\u00fcn mesajlar\u0131n t\u0259krar emal \u00fc\u00e7\u00fcn itirilm\u0259sin\u0259 s\u0259b\u0259b olmur, yaln\u0131z strategiyan\u0131n yenid\u0259n ba\u015flamas\u0131na s\u0259b\u0259b olur. <\/p>\n
    <\/p>\n
    Bu yana\u015fma, \u00e7ox b\u00f6y\u00fck y\u00fck s\u0259b\u0259bind\u0259n m\u00f6vcud olmaya bil\u0259n xarici sistemin y\u00fck\u00fcn\u00fc azaltma\u011fa k\u00f6m\u0259k edir. Ba\u015fqa s\u00f6zl\u0259, t\u0259krar emala \u0259lav\u0259 olaraq, biz circuit breaker<\/a><\/noindex>.<\/p>\n
    <\/p>\n
    Bizim hal\u0131m\u0131zda, s\u0259hv e\u015fikl\u0259rin say\u0131 c\u0259mi 1-dir v\u0259 m\u00fcv\u0259qq\u0259ti \u015f\u0259b\u0259k\u0259 kesintisi s\u0259b\u0259biyl\u0259 sistemin dayan\u0131ql\u0131\u011f\u0131n\u0131 minimaliz\u0259 etm\u0259k \u00fc\u00e7\u00fcn, biz ki\u00e7ik gecikm\u0259 intervallar\u0131 il\u0259 \u00e7ox detalla\u015fd\u0131r\u0131lm\u0131\u015f bir t\u0259krar \u00e7a\u011f\u0131rma strategiyas\u0131 istifad\u0259 edirik. Bu, \u015firk\u0259t qrupunun b\u00fct\u00fcn t\u0259tbiql\u0259ri \u00fc\u00e7\u00fcn uy\u011fun olmaya bil\u0259r, buna g\u00f6r\u0259 d\u0259 s\u0259hv e\u015fikl\u0259ri il\u0259 interval \u00f6l\u00e7\u00fcs\u00fc aras\u0131ndak\u0131 nisb\u0259t sistemin x\u00fcsusiyy\u0259tl\u0259rin\u0259 \u0259saslanaraq t\u0259nziml\u0259nm\u0259lidir.<\/p>\n
    <\/p>\n
    D\u0259yi\u015fk\u0259n m\u0259ntiq\u0259 sahib t\u0259tbiql\u0259rd\u0259n mesajlar\u0131 emal etm\u0259k \u00fc\u00e7\u00fcn ayr\u0131ca bir t\u0259tbiq<\/h3>\n
    <\/p>\n
    Bu t\u0259tbiq\u0259 (Retryer) mesaj g\u00f6nd\u0259rm\u0259k \u00fc\u00e7\u00fcn bir kod n\u00fcmun\u0259si:<\/p>\n
    <\/p>\n
    \npublic  void retry(ConsumerRecord record, String retryToTopic, \n                         Instant retryAt, String counter, String groupId, Exception e) {\n        Headers headers = ofNullable(record.headers()).orElse(new RecordHeaders());\n        List
     arrayOfHeaders = \n            new ArrayList(Arrays.asList(headers.toArray()));\n        updateHeader(arrayOfHeaders, GROUP_ID, groupId::getBytes);\n        updateHeader(arrayOfHeaders, DESTINATION, retryToTopic::getBytes);\n        updateHeader(arrayOfHeaders, ORIGINAL_PARTITION, \n                     () -> Integer.toString(record.partition()).getBytes());\n        if (nonNull(retryAt)) {\n            updateHeader(arrayOfHeaders, COUNTER, counter::getBytes);\n            updateHeader(arrayOfHeaders, SEND_TO, \"retry\"::getBytes);\n            updateHeader(arrayOfHeaders, RETRY_AT, retryAt.toString()::getBytes);\n        } else {\n            updateHeader(arrayOfHeaders, REASON, \n                         ExceptionUtils.getStackTrace(e)::getBytes);\n            updateHeader(arrayOfHeaders, SEND_TO, \"backout\"::getBytes);\n        }\n        ProducerRecord messageToSend =\n            new ProducerRecord(retryTopic, null, null, record.key(), record.value(), arrayOfHeaders);\n        kafkaTemplate.send(messageToSend);\n    }<\/code><\/pre>\n
    <\/p>\n
    N\u00fcmun\u0259d\u0259n g\u00f6r\u00fcnd\u00fcy\u00fc kimi, ba\u015fl\u0131qlarda \u00e7oxlu m\u0259lumat \u00f6t\u00fcr\u00fcl\u00fcr. RETRY_AT d\u0259y\u0259ri, istehsal\u00e7\u0131 dayand\u0131r\u0131lmas\u0131 yolu il\u0259 t\u0259krar mexanizmi \u00fc\u00e7\u00fcn oldu\u011fu kimi qal\u0131r. DESTINATION v\u0259 RETRY_AT-dan \u0259lav\u0259, biz \u00f6t\u00fcr\u00fcr\u00fck:<\/p>\n
    <\/p>\n