o fa @sUdZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z m Z m Z ddlmZmZmZmZmZddlmZddlmZddlmZdd lmZdd lmZeeZ d Z!d Z"d Z#dZ$dZ%dddZ&e e'e'fe(d<dddgegdZ)ee(d<dZ*dZ+dZ,dZ-dddZ.d d dZ/d!d"gZ0gd#Z1d$Z2de2iZ3 dxd%d&Z4d'e'd(ed)ed*e5d+df d,d-Z6d.d/Z7d0d1Z8d2d3Z9d4d5Z:d6d7Z;d8d9Zd>d?Z?d@e'd+e'fdAdBZ@d+e'fdCdDZAd+e'fdEdFZBdGdHZCdIe'd+eDfdJdKZEdLdMdNdOZFdPZGd+e e'e'ffdQdRZHdSdTZIdydVdWZJdXdYZKdzdZd[ZL d{d\d]ZMd^d_ZNd`daZOdbdcZPdddeZQdfdgZRdhdiZSdjdkZTdldmZUdndoZVdxdpdqZWdrdsZX   U td|dudvZYdwet | j }t |||t |ddr_t|||t||||t||zt|ttWnttfywtd Ynwd |vr|} || d <|d | d <t|d ||| |d dSdS)Nz#Nothing to do: No apt config and %szhandling apt config: %scodenamer/zApt Mirror info: %sadd_apt_repo_matchpreserve_sources_listFz)Failed to apply proxy or apt config info:sourcesRELEASEMIRROR)template_params aa_repo_match)rHLOGdebugr lsb_releaser(find_apt_mirror_infor<ADD_APT_REPO_MATCHrecompilesearch_ensure_dependenciesis_falseadd_mirror_keysgenerate_sources_listrename_apt_listsapply_apt_config APT_PROXY_FN APT_CONFIG_FNIOErrorOSError exceptionadd_apt_sources) r5r6r should_configmsgreleaser0mirrorsmatchermatchcfgparamsr1r1r2rBsH            rBcCs(|ds |d7}tjdg|dddS)N zdebconf-set-selectionsTdatacapture)endswithr ) selectionsr1r1r2debconf_set_selectionss  rrcCsg}g}|D]}|tvrtd|t|||q||qt|r-td|t|rAtjddgt|ddddSdS)Nzunconfiguring %szSThe following packages were installed and preseeded, but cannot be unconfigured: %szdpkg-reconfigurez--frontend=noninteractiveTrm)CONFIG_CLEANERSrQrRappendlenwarningr list)packages unhandled to_configpkgr1r1r2dpkg_reconfigures,     r|c s|dstddSdfddtD}t|t} D]\}}| D]}| dr;q3t dd |}||q3q+t}td |||}t|d kretd dSt|dS) z2apply_debconf_selections - push content to debconfdebconf_selectionsz(debconf_selections was not set in configN csg|]}|qSr1r1).0keyselsetsr1r2 sz,apply_debconf_selections..#z[:\s].*z pkgs_cfgd: %srzno need for reconfig)r<rQrRjoinsortedkeysrrencodesetitems splitlines startswithrVsubaddr get_installed_packages intersectionrur|) r5rq pkgs_cfgd_keycontentliner{pkgs_installed need_reconfigr1rr2rAs*            rAcCs6ttjdd}td||D]}t|qdS)z%clean out any local cloud-init configz/etc/cloud/cloud.cfg.d/*dpkg*pathz#cleaning cloud-init config from: %sN)globr target_pathrQrRosunlink)flistdpkg_cfgr1r1r2clean_cloud_inits   rcCsL|}|dr |dd}|d}|dkr||dd}|dd}|S)zmirrorurl_to_apt_fileprefix Convert a mirror url to the file prefix used by apt on disk to store cache information for that mirror. To do so do: - take off ???:// - drop tailing / - convert in string / to _/rz://N_)rpfindreplace)mirrorstringposr1r1r2mirrorurl_to_apt_fileprefixs    rc Cst|}tt}|D]\\}}||}|sq |tjjt |}|tjjt |}||kr2q t |} t d|D]+} d|| | df} t d| | zt| | Wq=tyht jdddYq=wq dS)z>rename_apt_lists - rename apt lists to preserve old cache dataz%s_*z%s%sNzRenaming apt list %s to %szFailed to rename apt list:T)exc_info)r3r r APT_LISTSrr<rrseprrurrQrRrenamerbrv) new_mirrorsr0default_mirrorsprer4omirrornmirroroprefixnprefixolenfilenamenewnamer1r1r2r]s*   r]cCs8dddddd}z||}W|Sty|}Y|Sw)zthere are a few default names which will be auto-extended. This comes at the inability to use those names literally as suites, but on the other hand increases readability of the cfg quite a lotz$RELEASE-updatesz$RELEASE-backportsz$RELEASE-securityz$RELEASE-proposedz$RELEASE)updates backportssecurityproposedrg)KeyError)suitemappingretsuiter1r1r2map_known_suites/s  r deb822_entrycCs<td|stdd|}ttd|}dt|dS|S)z0If no active Suites, disable this deb822 source.z\nSuites:[ \t]+([\w-]+)z \nSuites:.*rz7## Entry disabled by cloud-init, due to disable_suites z# disabled by cloud-init: )rVfindallrDISABLE_SUITES_REDACT_PREFIXr)rr1r1r2%disable_deb822_section_without_suitesAs rc s"g}fdd|Dtd|d}|D]j}|dr/|r)||d7}n||q|r5|rF|r@|t|d}||q|}|dsT||d7}qr||dd }fd d|D}||kr||t|d7}d d |}||d7}q|r|t|d |S) z:reads the deb822 format config and comment disabled suitescs g|] }tt|diqS)rM)r render_stringrrr)rgr1r2rTsz)disable_suites_deb822..zDisabling suites %s as %srrr~zSuites:Ncsg|]}|vr|qSr1r1r)disabled_suite_namesr1r2rrs zSuites:  ) rQrRrrrtisspacersplitrr) disabledsrcrgnew_srcnew_deb822_entryrnew_line orig_suites new_suitesr1)rrgr2disable_suites_deb822QsH          rc Cs|s|S|}t|rt|||S|D]`}t|}t|d|i}td||d}|dD]A}|dr:||7}q.| }t |dkrkd} |ddra|dd D] } | d7} | d r`nqS|| |krkd |}||7}q.|}q|S) zRreads the config for suites to be disabled and removes those from the templaterMzDisabling suite %s as %srTrr[N]z"# suite disabled by cloud-init: %s) is_deb822_sources_formatrrr rrQrRrrrrurp) rrrgretsrcr releasesuitenewsrcrcolspcolcolr1r1r2disable_suitess8      rcCs0dD]}||gD] }t||||dq qdS)z=Adds any keys included in the primary/security mirror clausesprimaryr) file_nameN)r< add_apt_key)r5r6rrrr1r1r2r[s r[apt_src_contentcCs6td|tjr dStd|tjrdStddS)a?Simple check for deb822 format for apt source content Only validates that minimal required keys are present in the file, which indicates we are likely deb822 format. Doesn't handle if multiple sections all contain deb822 keys. Return True if content looks like it is deb822 formatted APT source. z^(deb |deb-src )Fz'^(Types: |Suites: |Components: |URIs: )Tzapt.sources_list value does not match either deb822 source keys or deb/deb-src list keys. Assuming APT deb/deb-src list format.)rVrMrQrv)rr1r1r2rs rzetc/apt sources.listzsources.list.d)Dir::EtcDir::Etc::sourcelistDir::Etc::sourcepartsz@(Dir::Etc|Dir::Etc::sourceparts|Dir::Etc::sourcelist) \"([^\"]+)c Cs2z%ddl}||jdtd}|jdtd}|jdtd}Wnatyz tddg\}}Wn*tjyatd}td}td}d|d|d|d|dd YYSwt t |}t |}|dtd}|dtd}|dtd}Ynwd|d|d|d|dd S) aReturn a dict of applicable apt configuration or defaults. Prefer python apt_pkg if present. Fallback to apt-config dump command if present out output parsed Fallback to DEFAULT_APT_CFG if apt-config command absent or output unparsable. rNrrrz apt-configdumpr) sourcelist sourceparts) apt_pkg init_configconfigr<DEFAULT_APT_CFG ImportErrorr ProcessExecutionErrorrVr APT_CFG_REr>)retcrrapt_dumpr matched_cfgapt_cmd_configr1r1r2 get_apt_cfgsD  rcCst}|d}|d|jjd}tjr|}n|}||d}|D]} || || <|| || <q!|dd} | shtdtjrDdnd } | d |jj| } | sY| d } | sct d |dSt | } t | |} | rt| r||krtd ||}ntd||}t|d| |}t j||dd||krtj|rt|jj}|r|t |krtd|t |tdSdStd|t |dSdSdS)zgenerate_sources_list create a source.list file based on a custom or default template by replacing mirrors and release in the templaterrz.sources)rMrI sources_listNz1No custom template provided, fall back to builtinz.deb822rz sources.list.rz#No template found, not rendering %szAProvided 'sources_list' user-data is deb822 format, writing to %szFProvided 'sources_list' user-data is not deb822 format, fallback to %sri)modez*Replacing %s to favor deb822 source formatz)Removing %s to favor deb822 source format)rdistror4rAPT_DEB822_SOURCE_LIST_FILElowerr<rQinfoget_template_filenamervr load_text_filer rrrRr write_filerrexistsDEB822_ALLOWED_APT_SOURCES_LISTUBUNTU_DEFAULT_APT_SOURCES_LISTdel_file)r5rgrhr6rCapt_sources_listapt_sources_deb822 aptsrc_filerkktmpltmpl_fmt template_fnrenderedrexpected_contentr1r1r2r\sv         r\FcCsJtd|zt|j}td||||dWStjy$tdw)zM actual adding of a key as defined in key argument to the system zAdding key: '%s'r) output_filernhardenedz(failed to add apt GPG Key to apt keyring) rQrRpathlibPathstemapt_keyr rrc)rrrrr4r1r1r2add_apt_key_rawLs    rc Csg}t}t|ddr+dD]}||r*||D]}ddh|r)|dqq|di}|D]}ddh|rC|d||dd rP|d q5|D]} t| sa| t | qS|rn|j t |d Sd S) aInstall missing package dependencies based on apt_sources config. Inspect the cloud config user-data provided. When user-data indicates conditions where add_apt_key or add-apt-repository will be called, ensure the required command dependencies are present installed. Perform this inspection upfront because it is very expensive to call distro.install_packages due to a preliminary 'apt update' called before package installation. rKFrrkeyidrrLsourcerrN)rr rZr<rrvaluesshutilrGrtrrinstall_packagesr) r5rPr6missing_packages required_cmds mirror_key mirror_itemapt_sources_dictentcommandr1r1r2rY\s.         rYcCs`d|vrd|vrt}d|vr|d}||d||d<d|vr.t|d|p)|d||dSdS)z Add key to the system as defined in ent (if any). Supports raw keys or keyid's The latter will as a first step fetched to get the raw key rr keyserverrrN)DEFAULT_KEYSERVER getkeybyidr)r r6rrrr"r1r1r2r~src Cs|duri}|durtdt|tstd||D]}||}td|d|vr/||d<d|vrFd|dvrFt|||dd }||d <nt|||d|vrQq|d}t||}|d d smt j d |d|d<|d d s||dd 7<||rz tdd|gWntjytdwqtj|dd} zd|} d} d|vr|dsd} tj| | | dWqty} ztd| | d} ~ ww|jjdddS)a install keys and repo source .list files defined in 'sources' for each 'source' entry in the config: 1. expand template variables and write source .list file in /etc/apt/sources.list.d/ 2. install defined keys 3. update packages via distro-specific method (i.e. apt-key update) @param srcdict: a dict containing elements required @param cloud: cloud instance object Example srcdict value: { 'rio-grande-repo': { 'source': 'deb [signed-by=$KEY_FILE] $MIRROR $RELEASE main', 'keyid': 'B59D 5F15 97A5 04B7 E230 6DCA 0620 BBCF 0368 3F77', 'keyserver': 'pgp.mit.edu' } } Note: Deb822 format is not supported Nz did not get a valid repo matcherzunknown apt format: %szadding source/key '%s'rrz $KEY_FILETr#KEY_FILErz/etc/apt/sources.list.d/z.listrz --no-updatezadd-apt-repository failed.rz%s artw)omodezfailed write to file %s: %s)force)r.r=r> TypeErrorrQrRrr rrrrrrpr rrcrr rrarupdate_package_sources) srcdictr6rrOrPrr key_filersourcefncontentsr)detailr1r1r2rdsb       rdcCs~i}tjddddt|tr2td|D]}d|vr'd|d<t|d}n|d}|||<q|St|tr;|}|St d) z1convert v1 apt format to v2 (dict in apt_sources)zConfig key 'apt_sources'22.1zUse 'apt' instead) deprecateddeprecated_version extra_messagez9apt config: convert V1 to V2 format (source list to dict)rzcloud_config_sources.listzunknown apt_sources format) r deprecater=rwrQrRr rand_dict_keyr>r.)srclistr-srcentrr1r1r2convert_v1_to_v2_apt_formats&    r:cCs,||ddur||||<||=dSdS)ziconvert an old key to the new one if the old one exists returns true if a key was found and convertedNTFr<)oldcfgaptcfgoldkeynewkeyr1r1r2 convert_keys r@cCsLgd}d}ddgi}|D] \}}t||||rd}q |r$|g|d<dSdS)zBconvert old apt_mirror keys into the new more advanced mirror spec)) apt_mirroruri)apt_mirror_searchrX)apt_mirror_search_dns search_dnsFarchesdefaultTrN)r@)r<r=keymap convertednewmcfgr>r?r1r1r2convert_mirrors  rKc CsVddddddddddd d }g}|D]}||vr'||d vr"||=q||q|s,|Stjd |d d|dd}|durwtjdd d|D]+}||}||}||=|dusb||ddurcqI|||krttd||||fqI|Si}|D]}||durt|||||q{t|||D]}||ddurtd|q||d<|S)z:convert old to new keys and adapt restructured mirror specrLNproxy http_proxy https_proxy ftp_proxyrKrrJ) apt_sourcesrArCrD apt_proxyapt_http_proxy apt_ftp_proxyapt_https_proxyapt_preserve_sources_listapt_custom_sources_listrJ)NrzThe following config key(s): r2)r3r4r9z0Support for combined old and new apt module keysz@Old and New apt format defined with unequal values %s vs %s @ %sz&old apt key '%s' left after conversion)rtrr6r<r.r@rK)r< mapoldkeys needtoconvertr> newaptcfgr?verifyr=r1r1r2convert_v2_to_v3_apt_format!sr         r[cCs,|dd}|durt||d<t|}|S)zconvert the old list based format to the new dict based one. After that convert the old dict keys/format to v3 a.k.a 'new apt config'rPN)r<r:r[)r5rPr1r1r2r;ks  r;c Csd}|rZd}g}|dkrd}n |dkrd}ntdt||j}d|dd d}|r6|d ||d g} |jj } d | |d | f} |D] } | | | qKt | }|S)zG Try to resolve a list of predefines DNS names to pick mirrors Nrrrrzsecurity-mirrorzunknown mirror type.rz.%s)z .localdomainrzhttp://%s-%s%s/%sz%s) r.r get_hostname_fqdnfqdnrrrtextendrr4search_for_mirror) configured mirrortyper5r6rmydomdoms mirrordnsr^ mirror_listr mirrorfmtpostr1r1r2search_for_mirror_dnszs. ricCsX|dur|dur |}||dS|j}|r(|}|d|d<|d|d<|St|S)z^sets security mirror to primary if not defined. returns defaults if no mirrors are definedNrrrrr) datasourceget_package_mirror_infor+r3)pmirrorsmirrorr0r6 mirror_infomr1r1r2update_mirror_infos    rpcCsT||d}|dur dSd}|D]}|dpg}||vr!|Sd|vr'|}q|S)zuout of a list of potential mirror configurations select and return the one matching the architecture (or default)NrFrGr;)r5rbr0mirror_cfg_listrGmirror_cfg_elemrFr1r1r2get_arch_mirrorconfigs rscCs`t|||}|dur dS|dd}|durt|dd}|dur.t|dd|||}|S)zpass the three potential stages of mirror specification returns None is neither of them found anything otherwise the first hit is returnedNrBrXrE)rsr<r r`ri)r5rbr0r6mcfgrr1r1r2 get_mirrors  rucCsn|durt}td|t|d||}td|t|d||}td|t||||}|d|d<|S) afind_apt_mirror_info find an apt_mirror given the cfg provided. It can check for separate config of primary and security mirrors If only primary is given security is assumed to be equal to primary If the generic apt_mirror is given that is defining for both Nz!got arch for mirror selection: %srzgot primary mirror: %srzgot security mirror: %srrN)r r(rQrRrurp)r5r6r0rlrmrnr1r1r2rTs    rTcsd}fdd|D}t|r!td|t|d|dntj|r2t |td| ddrItd |t| ddStj|r\t |td |dSdS) zHapply_apt_config Applies any apt*proxy config from if specified ))rLAcquire::http::Proxy "%s";)rMrv)rOzAcquire::ftp::Proxy "%s";)rNzAcquire::https::Proxy "%s";cs(g|]\}}|r||qSr1r;)rr4fmtr5r1r2r s(z$apply_apt_config..zwrite apt proxy info to %sr~z#no apt proxy configured, removed %sconfNzwrite apt config info to %sz$no apt config configured, removed %s) rurQrRr rrrrisfilerr<)r5 proxy_fname config_fnamecfgsproxiesr1rxr2r^s        r^TcsVddfdd}fdd}|dkr||S|dks#|d kr'||Std ) adapt-key replacement commands implemented: 'add', 'list', 'finger' @param output_file: name of output gpg file (without .gpg or .asc) @param data: key contents @param human_output: list keys formatted for human parsing @param hardened: write keys to to /etc/apt/cloud-init.gpg.d/ (referred to with [signed-by] in sources file) cSsJtjtr tgng}ttD]}|dr|t|q|r#|SdS)zreturn all apt keys /etc/apt/trusted.gpg (if it exists) and all keyfiles (and symlinks to keyfiles) in /etc/apt/trusted.gpg.d/ are returned based on apt-key implementation )z.gpgz.ascr)rrrzAPT_LOCAL_KEYSlistdirAPT_TRUSTED_GPG_DIRrprt) key_filesfiler1r1r2_get_key_files/s   zapt_key.._get_key_filesc sd}sttd|Szrtnt}|}d|}t||W|Stj y=ttdY|St yOttdY|Sw)ziapt-key add returns filepath to new keyring, or '/dev/null' when an error occurs z /dev/nullz)Unknown filename, failed to add key: "{}"z{}{}.gpgz Gpg error, failed to add key: {}z#Decode error, failed to add key: {}) r logexcrQr?CLOUD_INIT_GPG_DIRrdearmorrr rUnicodeDecodeError)rDrkey_dirstdout)rnrrr1r2 apt_key_add>s0        zapt_key..apt_key_addc sfg}D](}z ||j|dWqtjy-}z td||WYd}~qd}~wwd|S)zapt-key list returns string of all trusted keys (in /etc/apt/trusted.gpg and /etc/apt/trusted.gpg.d/) ) human_outputzFailed to list key "%s": %sNr~)rt list_keysr rrQrvr)rDkey_listr.error)rrr1r2 apt_key_listZs   zapt_key..apt_key_listrfingerrwz@apt_key() commands add, list, and finger are currently supported)r.)r!rrrnrrrrr1)rrnrrrr2rsrz cloud-init)N)F)FN)NN)NNFT)[__doc__rloggingrrrVrtextwraprtypingrrrr cloudinitrrr r r cloudinit.cloudr cloudinit.configr cloudinit.config.schemar cloudinit.gpgrcloudinit.settingsr getLogger__name__rQrUrrrrrstr__annotations__rrr`r_r$r*r-r)r,rrr3rwrErHrBrrr|rArrr]rrrrr[boolrrrrr\rrYrrdr:r@rKr[r;rirprsrurTr^rrsr1r1r1r2s           .  # /(3 F " X J)  Y