A Pilot Study of Combined Immune Checkpoint Inhibition in Combination With Ablative Therapies in Subjects With Hepatocellular Carcinoma (HCC) or Biliary Tract Carcinomas (BTC)

* INCLUSION CRITERIA: 1. Patients must have histopathological confirmation of hepatocellular carcinoma (HCC) or biliary tract carcinoma (BTC) by the Laboratory of Pathology of the National Cancer Institute (NCI) prior to entering this study OR histopathological confirmation of carcinoma in the setting of clinical and radiological characteristics which, together with the pathology, are highly suggestive of a diagnosis of HCC (or biliary tract carcinoma). Fibrolamellar variant is also allowed. The term BTC includes intra or extrahepatic cholangiocarcinoma, gallbladder cancer or ampullary cancer. 2. Patients must have disease that is not amenable to potentially curative resection, transplantation or ablation. HCC patients must have progressed on, been intolerant to, or refused prior sorafenib therapy. Patients with BTC must have received, been intolerant of or refused at least one line of chemotherapy. 3. Patients must have multiple tumor lesions (at least 2): one for the ablation procedure and another for evaluation located outside the proposal ablation zone. 4. Disease must be technically amenable to transhepatic arterial chemoembolization (TACE) (HCC patients only), radiofrequency ablation (RFA), or cryoablation. Each case will be discussed at gastrointestinal (GI) tumor board with interventional radiology. Patients must have evaluable disease. 5. If liver cirrhosis is present, patient must have a Child-Pugh A/B7 classification 6. Age 18 years or older 7. Eastern Cooperative Oncology Group (ECOG) performance status 0-2 8. Patients must have normal organ and marrow function as defined below: * leukocytes greater than or equal to 3,000/mcL * absolute neutrophil count greater than or equal to 1,000/mcL * platelets greater than or equal to 60,000/mcL * total bilirubin If cirrhosis present: Part of Child Pugh requirement; If no cirrhosis: bilirubin should be less than or equal to 2 x upper limit of normal (ULN) * serum albumin If cirrhosis present: Part of Child Pugh requirement; If no cirrhosis: albumin should be less than or equal to 2.5g/dl * patients are eligible with alanine aminotransferase (ALT) or aspartate aminotransferase (AST) up to 5 x ULN. * creatinine \< 1.5x institution upper limit of normal OR creatinine clearance greater than or equal to 45 mL/min/1.73 m\^2 for patients with creatinine levels above institutional normal 9\. Patients must have recovered from any acute toxicity related to prior therapy, including surgery. Toxicity should be less than or equal to grade 1 or returned to baseline. 10\. Patients must not have other invasive malignancies within the past 5 years (with the exception of non-melanoma skin cancers, non-invasive bladder cancer or localized prostate cancer for whom systemic therapy is not required). 11\. Patient must be able to understand and willing to sign a written informed consent document. 12\. Evidence of post-menopausal status or negative urinary or serum pregnancy test for female pre-menopausal patients. Women will be considered post-menopausal if they have been amenorrheic for 12 months without an alternative medical cause. The following age-specific requirements apply: * Women \<50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of exogenous hormonal treatments and if they have luteinizing hormone and follicle-stimulating hormone levels in the post-menopausal range for the institution or underwent surgical sterilization (bilateral oophorectomy or hysterectomy). * Women greater than or equal to 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of all exogenous hormonal treatments, had radiation-induced menopause with last menses \>1 year ago, had chemotherapy-induced menopause with last menses \>1 year ago, or underwent surgical sterilization (bilateral oophorectomy, bilateral salpingectomy or hysterectomy. Subject is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations including follow up. 13\. Body weight greater than 30kg EXCLUSION CRITERIA: 1. Patients who have had standard of care chemotherapy, large field radiotherapy, or major surgery must wait 2 weeks prior to entering the study. For recent experimental therapies a 28 day period of time must elapse before treatment. 2. Patients who have undergone prior liver transplantation are ineligible. 3. Patients with known brain metastases will be excluded from this clinical trial because of their poor prognosis and because they often develop progressive neurologic dysfunction that would confound the evaluation of neurologic and other adverse events. 4. Uncontrolled intercurrent illness including, but not limited to, ongoing or active systemic infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia (excluding insignificant sinus bradycardia and sinus tachycardia) or psychiatric illness/social situations that would limit compliance with study requirements. 5. History of chronic autoimmune disease (e.g., Addison's disease, multiple sclerosis, Graves disease, Hashimoto's thyroiditis, rheumatoid arthritis, hypophysitis, etc.) with symptomatic disease within the 3 years before randomization. Note: Active vitiligo or a history of vitiligo will not be a basis for exclusion. 6. Dementia or significantly altered mental status that would prohibit the understanding or rendering of Information and Consent and compliance with the requirements of the protocol. 7. Diverticulitis either active or history of within the past 2 years. Note that diverticulosis is permitted. 8. Active or history of inflammatory bowel disease (colitis, Crohn's), irritable bowel disease, celiac disease, or other serious, chronic, gastrointestinal conditions associated with diarrhea. Active or history of systemic lupus erythematosus or Wegener's granulomatosis. Currently receiving immunosuppressive doses of steroids or other immunosuppressive medications (inhaled and topical steroids are permitted) 9. History of sarcoidosis syndrome 10. Patients should not be vaccinated with live attenuated vaccines within 30 days of starting durvalumab or tremelimumab treatment. 11. Has a known history of Human Immunodeficiency Virus (HIV). HIV-positive patients receiving anti-retroviral therapy are excluded from this study due to the possibility of pharmacokinetic interactions between antiretroviral medications and tremelimumab. HIV positive patients not receiving antiretroviral therapy are excluded due to the possibility that tremelimumab may worsen their condition and the likelihood that the underlying condition may obscure the attribution of adverse events. 12. History of hypersensitivity reaction to human or mouse antibody products. 13. Pregnancy and breast feeding are exclusion factors. The effects of tremelimumab on the developing human fetus are unknown. Enrolled patients must agree to use adequate contraception prior to study entry, the duration of study participation and 6 months after the end of the treatment. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. 14. Patients with unhealed surgical wounds for more than 30 days.