f9dZddlZddlZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZdZd ZGd d ZGd d eZGddeZy)u:class:`.RateLimiter` and :class:`.AsyncRateLimiter` allow to perform bulk operations while gracefully handling error responses and adding delays when needed. In the example below a delay of 1 second (``min_delay_seconds=1``) will be added between each pair of ``geolocator.geocode`` calls; all :class:`geopy.exc.GeocoderServiceError` exceptions will be retried (up to ``max_retries`` times):: import pandas as pd df = pd.DataFrame({'name': ['paris', 'berlin', 'london']}) from geopy.geocoders import Nominatim geolocator = Nominatim(user_agent="specify_your_app_name_here") from geopy.extra.rate_limiter import RateLimiter geocode = RateLimiter(geolocator.geocode, min_delay_seconds=1) df['location'] = df['name'].apply(geocode) df['point'] = df['location'].apply(lambda loc: tuple(loc.point) if loc else None) This would produce the following DataFrame:: >>> df name location \ 0 paris (Paris, Île-de-France, France métropolitaine, ... 1 berlin (Berlin, 10117, Deutschland, (52.5170365, 13.3... 2 london (London, Greater London, England, SW1A 2DU, UK... point 0 (48.8566101, 2.3514992, 0.0) 1 (52.5170365, 13.3888599, 0.0) 2 (51.5073219, -0.1276474, 0.0) To pass extra options to the `geocode` call:: from functools import partial df['location'] = df['name'].apply(partial(geocode, language='de')) To see a progress bar:: from tqdm import tqdm tqdm.pandas() df['location'] = df['name'].progress_apply(geocode) Before using rate limiting classes, please consult with the Geocoding service ToS, which might explicitly consider bulk requests (even throttled) a violation. N)chaincount)sleep default_timer)GeocoderServiceError)logger)AsyncRateLimiter RateLimiterc<tdt|DdgS)z-list(_is_last_gen(2)) -> [False, False, True]c3 K|]}dyw)FN).0_s ?D:\switchATM\venv\Lib\site-packages\geopy/extra/rate_limiter.py z_is_last_gen..Bs.A%.s T)rrange)rs r _is_last_genr@s .u. 77c4eZdZdZefZdZdZdZdZ dZ y)BaseRateLimiterz9Base Rate Limiter class for both sync and async versions.c||_||_||_||_|dk\sJt j |_d|_yNr)min_delay_seconds max_retriesswallow_exceptionsreturn_value_on_exception threadingLock_lock _last_call)selfrrrrs r__init__zBaseRateLimiter.__init__JsF"3&"4)B&a^^% rctSNr)r"s r_clockzBaseRateLimiter._clock\s rc#K |j5|j}|j||_ dddy||jz }|j|z }|dkr||_ dddy ddd|#1swYxYwwr)r r&r!r)r"clockseconds_since_last_callwaits r_acquire_request_slot_genz)BaseRateLimiter._acquire_request_slot_gen_s*  ??*&+DO   +0$//*A'--0GG19&+DO   J  s,B %B B +B) B 3 B B B c #0Kttt|jD] \}} |yy#|j$rK|rdnAt j t|jdz||j||ddYfYhwxYww)NTzB caught an error, retrying (%s/%s tries). Called with (*%r, **%r).exc_infoF) ziprrr_retry_exceptionsr warningtype__name__)r"argskwargsi is_last_trys r _retries_genzBaseRateLimiter._retries_gens!%'<8H8H+IJ NA{ $) )) JNNT ++/BB((!% K s&.B9BABBBBc|jrFtjt|jdz|j ||d|j S)Nz> swallowed an error after %r retries. Called with (*%r, **%r).Tr-)rr r1r2r3rr)r"r4r5s r _handle_exczBaseRateLimiter._handle_excsQ  " " NNT ##'++    11 1 rN) r3 __module__ __qualname____doc__rr0r#r&r+r8r:rrrrrEs(C-/$$L. rrcBeZdZdZddddddfd Zd Zd Zd ZxZS) r aThis is a Rate Limiter implementation for synchronous functions (like geocoders with the default :class:`geopy.adapters.BaseSyncAdapter`). Examples:: from geopy.extra.rate_limiter import RateLimiter from geopy.geocoders import Nominatim geolocator = Nominatim(user_agent="specify_your_app_name_here") search = ["moscow", "paris", "berlin", "tokyo", "beijing"] geocode = RateLimiter(geolocator.geocode, min_delay_seconds=1) locations = [geocode(s) for s in search] search = [ (55.47, 37.32), (48.85, 2.35), (52.51, 13.38), (34.69, 139.40), (39.90, 116.39) ] reverse = RateLimiter(geolocator.reverse, min_delay_seconds=1) locations = [reverse(s) for s in search] RateLimiter class is thread-safe. If geocoding service's responses are slower than `min_delay_seconds`, then you can benefit from parallelizing the work:: import concurrent.futures geolocator = OpenMapQuest(api_key="...") geocode = RateLimiter(geolocator.geocode, min_delay_seconds=1/20) with concurrent.futures.ThreadPoolExecutor() as e: locations = list(e.map(geocode, search)) .. versionchanged:: 2.0 Added thread-safety support. @TNrrerror_wait_secondsrrcdt|||||||_||_||k\sJ|dk\sJya  :param callable func: A function which should be wrapped by the rate limiter. :param float min_delay_seconds: Minimum delay in seconds between the wrapped ``func`` calls. To convert :abbr:`RPS (Requests Per Second)` rate to ``min_delay_seconds`` you need to divide 1 by RPS. For example, if you need to keep the rate at 20 RPS, you can use ``min_delay_seconds=1/20``. :param int max_retries: Number of retries on exceptions. Only :class:`geopy.exc.GeocoderServiceError` exceptions are retried -- others are always re-raised. ``max_retries + 1`` requests would be performed at max per query. Set ``max_retries=0`` to disable retries. :param float error_wait_seconds: Time to wait between retries after errors. Must be greater or equal to ``min_delay_seconds``. :param bool swallow_exceptions: Should an exception be swallowed after retries? If not, it will be re-raised. If yes, the ``return_value_on_exception`` will be returned. :param return_value_on_exception: Value to return on failure when ``swallow_exceptions=True``. )rrrrrNsuperr#funcrCr"rHrrrCrr __class__s rr#zRateLimiter.__init__OR /#1&?   "4!%666arcrtjt|jdz|t |yNz sleep(%r))r debugr2r3rr"secondss r_sleepzRateLimiter._sleeps& T$Z((<7A grcP|jD]}|j|yr%r+rQr"r*s r_acquire_request_slotz!RateLimiter._acquire_request_slot s&224 D KK  rc|j||}|D]H}|j |j|i|}tj|r t d|cStd#|j $rO}|j|r|j||cYd}~cS|j|jYd}~d}~wwxYw)NzoAn async awaitable has been passed to `RateLimiter`. Use `AsyncRateLimiter` instead, which supports awaitables.Should not have been reached) r8rUrHinspect isawaitable ValueErrorr0throwr:rQrC RuntimeError)r"r4r5genrreses r__call__zRateLimiter.__call__ sf- 5A  & & ( 5dii00&&s+$U  5 9:: )) 599Q<++D&99 D3344  5s#3A++C :"CC $CC  r3r;r<r=r#rQrUr` __classcell__rJs@rr r s/#R"&2 h;rr cBeZdZdZddddddfd Zd Zd Zd ZxZS) r aThis is a Rate Limiter implementation for asynchronous functions (like geocoders with :class:`geopy.adapters.BaseAsyncAdapter`). Examples:: from geopy.adapters import AioHTTPAdapter from geopy.extra.rate_limiter import AsyncRateLimiter from geopy.geocoders import Nominatim async with Nominatim( user_agent="specify_your_app_name_here", adapter_factory=AioHTTPAdapter, ) as geolocator: search = ["moscow", "paris", "berlin", "tokyo", "beijing"] geocode = AsyncRateLimiter(geolocator.geocode, min_delay_seconds=1) locations = [await geocode(s) for s in search] search = [ (55.47, 37.32), (48.85, 2.35), (52.51, 13.38), (34.69, 139.40), (39.90, 116.39) ] reverse = AsyncRateLimiter(geolocator.reverse, min_delay_seconds=1) locations = [await reverse(s) for s in search] AsyncRateLimiter class is safe to use across multiple concurrent tasks. If geocoding service's responses are slower than `min_delay_seconds`, then you can benefit from parallelizing the work:: import asyncio async with OpenMapQuest( api_key="...", adapter_factory=AioHTTPAdapter ) as geolocator: geocode = AsyncRateLimiter(geolocator.geocode, min_delay_seconds=1/20) locations = await asyncio.gather(*(geocode(s) for s in search)) .. versionadded:: 2.0 r?r@rATNrBcdt|||||||_||_||k\sJ|dk\sJyrErFrIs rr#zAsyncRateLimiter.__init__LrKrcKtjt|jdz|t j |d{y7wrM)r rNr2r3asynciorrOs rrQzAsyncRateLimiter._sleeps5 T$Z((<7AmmG$$$sAAA AclK|jD]}|j|d{y7wr%rSrTs rrUz&AsyncRateLimiter._acquire_request_slots3224 $D++d# # # $ #s (424cK|j||}|D]6}|jd{ |j|i|d{cStd7-7#|j$rX}|j |r|j ||cYd}~cS|j |jd{7Yd}~d}~wwxYww)NrW) r8rUrHr0r[r:rQrCr\)r"r4r5r]rr_s rr`zAsyncRateLimiter.__call__sf- ;A,,. . . ;&TYY7777 ;9:: /7)) ;99Q<++D&99kk$"9"9:::  ;sh+C AC AA A C AC."CCC C6B97C<C CC rarcs@rr r "s/'Z"&2 h%$ ;rr )r=rgrXr itertoolsrrtimertimeitr geopy.excr geopy.utilr __all__rrr r rrrrpsY0d" * -8 ccLt;/t;nr;r;r