Pembrolizumab, a Monoclonal Antibody Against PD-1, in Combination With Capecitabine and Oxaliplatin (CAPOX) in People With Advanced Biliary Tract Carcinoma (BTC)

* INCLUSION CRITERIA: * Patients must have histopathological confirmation of biliary tract carcinoma (BTC) by the Laboratory of Pathology of the National Cancer Institute (NCI) prior to entering this study OR histopathological confirmation of carcinoma in the setting of clinical and radiological characteristics which, together with the pathology, are highly suggestive of a diagnosis of biliary tract carcinoma. The term BTC includes intra- or extrahepatic cholangiocarcinoma, gallbladder cancer or ampullary cancer. * Patients must have disease that is not amenable to potentially curative resection. Patients must have received, been intolerant of or refused at least one line of chemotherapy. * Patients must have at least one focus of measurable metastatic disease per Response Evaluation Criteria in Solid Tumors (RECIST) 1.1. * Patients must have at least one focus of metastatic disease that is amenable to pre- and on-treatment biopsies. Ideally the biopsied lesion should not be one of the target measurable lesions, although this can be up to the discretion of the investigators. * Age greater than or equal to 18 years * Eastern Cooperative Oncology Group (ECOG) performance status 0-1 * Patients must have normal organ and marrow function as defined below: * leukocytes greater than or equal to 3,000/mcL * absolute neutrophil count greater than or equal to 1,000/mcL * platelets greater than or equal to 100,000/mcL * total bilirubin less than or equal to 2 xULN * Serum albumin greater than or equal to 2.5g/dl * Patients are eligible with alanine aminotransferase (ALT) or aspartate aminotransferase (AST) up to 5 x upper limit of normal (ULN). * creatinine \<1.5X institution upper limit of normal OR creatinine clearance greater than or equal to 45 mL/min/1.73 m\^2 for patients with creatinine levels above institutional normal * Patients must have recovered from any acute toxicity related to prior therapy, including surgery. Toxicity should be less than or equal to grade 1 or returned to baseline. * Patients must not have other invasive malignancies within the past 5 years (with the exception of non-melanoma skin cancers, non-invasive bladder cancer or localized prostate cancer for whom systemic therapy is not required). * Patient must be able to understand and willing to sign a written informed consent document. * The effects of Pembrolizumab in combination with Capecitabine and Oxaliplatin on the developing human fetus are unknown. For this reason, women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry, for the duration of study participation and up to 120 days after the last dose of the drug. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. EXCLUSION CRITERIA: * Patients who have had standard of care chemotherapy, large field radiotherapy, or major surgery must wait 2 weeks prior to entering the study. * Previous treatment with immune checkpoint inhibitors. * Patients who have undergone prior liver transplantation are ineligible. * Patients with known brain metastases will be excluded from this clinical trial because of their poor prognosis and because they often develop progressive neurologic dysfunction that would confound the evaluation of neurologic and other adverse events. * Uncontrolled intercurrent illness including, but not limited to, ongoing or active systemic infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia (excluding insignificant sinus bradycardia and sinus tachycardia) or psychiatric illness/social situations that would limit compliance with study requirements. * History of (non-infectious) pneumonitis that required steroids, evidence of interstitial lung disease or active, non-infectious pneumonitis. * History of chronic autoimmune disease (e.g., Addison s disease, multiple sclerosis, Graves disease, Hashimoto's thyroiditis, rheumatoid arthritis, hypophysitis, etc.) with symptomatic disease within the 3 years before randomization. Note: Active vitiligo or a history of vitiligo will not be a basis for exclusion. * Dementia or significantly altered mental status that would prohibit the understanding or rendering of Information and Consent and compliance with the requirements of the protocol * Active or history of inflammatory bowel disease (colitis, Crohn's), irritable bowel disease, celiac disease, or other serious, chronic, gastrointestinal conditions associated with diarrhea. Active or history of systemic lupus erythematosus or Wegener's granulomatosis. * Currently receiving immunosuppressive doses of steroids or other immunosuppressive medications (inhaled and topical steroids are permitted) * History of sarcoidosis syndrome. * Known history of active tuberculosis. * Patients should not be vaccinated with live attenuated vaccines within 1 month of starting pembrolizumab treatment. * Active hepatitis B or C infection. * Human Immunodeficiency Virus (HIV)-positive patients receiving anti-retroviral therapy are excluded from this study due to the possibility of pharmacokinetic interactions between antiretroviral medications and pembrolizumab. HIV positive patients not receiving antiretroviral therapy are excluded due to the possibility that pembrolizumab may worsen their condition and the likelihood that the underlying condition may obscure the attribution of adverse events. * History of hypersensitivity reaction to human or mouse antibody products. * Female patients who are pregnant or breastfeeding. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with Pembrolizumab in combination with Capecitabine and Oxaliplatin, breastfeeding should be discontinued. * Patients with unhealed surgical wounds for more than 30 days. * Prior therapy with oxaliplatin