The Purpose of This Trial is to Determine if Regorafenib Plus Durvalumab (MEDI4736) is Safe and Effective in Treatment of Chemo Refractory Advanced Biliary Tract Cancers

Inclusion Criteria: * Ability of patient OR Legally Authorized Representative (LAR) to understand this study, and participant or LAR willingness to sign a written informed consent * Can swallow tablets and self-administer medication * Progressed on at least one line of therapy (no restrictions on type of previous treatment) * Eastern Cooperative Oncology Group (ECOG) Performance Status = 0 - 1 * Measurable disease with at least 1 lesion that qualifies as a RECIST 1.1 Target Lesion (TL) at baseline. Previously irradiated lesion cannot be considered as Target Lesion (TL) except in cases of documented progression of the lesion since the completion of radiation therapy * Histologically confirmed unresectable or metastatic intrahepatic/extrahepatic cholangiocarcinoma or gallbladder cancer with radiographic progression, who have progressed on one line of therapy / failed adjuvant therapy * Life expectancy of at least 3 months * Recovery to baseline or \< Grade 2 CTCAE v5.0 from toxicities related to any prior treatments, unless adverse event's (AE(s)) are clinically nonsignificant and/or stable on supportive therapy * Adequate organ function per laboratory results * Concomitant anticoagulation with oral anticoagulants (e.g., warfarin, direct thrombin and Factor Xa inhibitors) or platelet inhibitors (e.g., clopidogrel) will be allowed provided that no prior evidence of underlying abnormality in coagulation parameters exists. Close monitoring of at least weekly evaluations will be performed until international normalized ratio/ partial thromboplastin time (INR/PTT) is stable based on a measurement that is pre-dose as defined by the local standard of care * Weight \> 30 kg (66 lbs) * Women of child-bearing potential and men with partners of child-bearing potential must agree to use an acceptable form of contraception for the duration of study participation, and for 7 months after the last study treatment * Men of child-bearing potential must agree not to donate sperm while on this study and for 180 days (6 months) after the last dose of study treatment Exclusion Criteria: * Current or anticipated use of other investigational agents while participating in another clinical study, unless it is an observational (noninterventional) clinical study or during the follow-up period of an interventional study * Psychiatric illness/social situations that would limit compliance with study requirements * Pregnant or breastfeeding * Ampullary carcinoma * Previous treatment with regorafenib * Previous treatment with a programmed death 1 (PD1), programmed death-ligand (PD-L1, including durvalumab), or cytotoxic T-lymphocyte-associated protein 4 (CTLA-4) inhibitors, or agent directed to another co-inhibitory T cell receptor * Previous treatment with live vaccine within 30 days of planned start of study drugs (seasonal flu vaccines that do not contain a live virus are permitted) * Active autoimmune disease (active defined as having autoimmune disease related symptoms and detectable autoantibodies) that has required systemic treatment in the past 2 years * Diagnosis of immunodeficiency or receiving chronic systemic steroid therapy or any other form of immunosuppressive therapy within 7 days prior to the first dose of study drugs. Except Intranasal, inhaled, topical steroids, or local steroid injections (e.g., intra articular injection), systemic corticosteroids at physiologic doses not to exceed 10 mg/day of prednisone or its equivalent, steroids as premedication for hypersensitivity reactions (e.g., CT scan premedication) * Known history of human immunodeficiency virus (HIV) infection (HIV 1/2 antibodies). Dual active hepatitis B virus (HBV) infection (HBsAg (+) and /or detectable HBV DNA) and HCV infection (anti-HCV Ab(+) and detectable HCV RNA) at study entry. Single active infection of HBV or HCV infection is allowed with treatment by local standards * Has untreated central nervous system (CNS) metastases and/or carcinomatous meningitis identified either on the baseline brain imaging, unless known and treated with stable for \>4 weeks * Significant acute gastrointestinal disorders with diarrhea as a major symptom e.g., Crohn's disease, malabsorption, or CTCAE Grade ≥ 2 diarrhea of any etiology * Receipt of the last dose of anticancer therapy (chemotherapy, immunotherapy, endocrine therapy, targeted therapy, biologic therapy, tumor embolization, monoclonal antibodies) ≤ 21 days prior to the first dose of study drug * Any concurrent chemotherapy, investigational product (IP), biologic, or hormonal therapy for cancer treatment. Concurrent use of hormonal therapy for non-cancer-related conditions (e.g., hormone replacement therapy) is acceptable. However, the palliative radiation to non-targeted lesions is allowed * Abdominal fistula, GI perforation, bowel obstruction, or intra-abdominal abscess within 8 weeks before first dose. Note: Complete healing of an intra-abdominal abscess must be confirmed before first dose * Uncontrollable ascites or pleural effusion * Clinically significant gross hematuria, hematemesis, or hemoptysis of \>0.5 tsp (2.5ml) of red blood, or other history of grade 3 significant bleeding within 8 weeks * Any unresolved toxicity NCI CTCAE v 5.0 Grade ≥2 from previous anticancer therapy with the exception of neuropathy grade 2 and below, alopecia, vitiligo, and the laboratory values defined in the inclusion criteria * Major surgery (e.g., GI surgery, removal or biopsy of brain metastasis) within 8 weeks before first dose of study treatment. Complete wound healing from major surgery must have occurred 1 month before first dose and from minor surgery (e.g., simple excision, tooth extraction) at least 10 days before first dose. Patients with clinically relevant ongoing complications from prior surgery are not eligible * History of organ transplantation * Uncontrolled intercurrent illness, including but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, interstitial lung disease * Uncontrolled hypertension defined as sustained blood pressure (BP) \> 150 mm Hg systolic or \> 100 mm Hg diastolic despite optimal antihypertensive treatment * Mean QT interval corrected for heart rate (QTcF) \>470 ms calculated from 3 electrocardiograms (ECGs) (within 15 minutes at 5 minutes apart) using Fridericia's Correction * Stroke (including transient ischemic attack transient ischemic attack (TIA), myocardial infarction (MI), or other ischemic event, or acute thromboembolic event (e.g., deep venous thrombosis, pulmonary embolism) that are NOT asymptomatic with local standard anti-coagulation within 4 weeks before first dose * History of another primary malignancy in the last 3 years except: * Malignancy treated with curative intent and with no known active disease ≥3 years before the first dose of IP and of low potential risk for recurrence * Adequately treated non-melanoma skin cancer or lentigo maligna without evidence of disease * Adequately treated carcinoma in situ without evidence of disease * Use of any Herbal remedy * Ongoing infection \>grade 2 * Known allergy or hypersensitivity to any of the study drugs * Proteinuria \> Grade3 (\>3.5g/24 hours) * Active infection with tuberculosis (TB) (clinical evaluation that includes clinical history, physical examination and radiographic findings, and TB testing in line with local practice) * Participants with HBV infection (as characterized by positive hepatitis B surface antigen \[HBsAg\] and/or anti-hepatitis B core antibodies (anti-HBc) with detectable HBV deoxyribonucleic acid (DNA) \[≥10 international units (IU)/mL or above the limit of detection per local laboratory\]) must receive antiviral therapy prior to randomization to ensure adequate viral suppression * Participants must remain on antiviral therapy for the study duration and for 6 months after the last dose of study treatment. Participants who test positive for anti-HBc with undetectable HBV DNA (\<10 IU/mL or under the limit of detection per local laboratory) do not require antiviral therapy unless HBV DNA exceeds 10IU/mL or reaches detectable limits per local laboratory during the course of treatment * Patients positive for hepatitis C (HCV) antibody. EXCEPTIONS: Patients positive for hepatitis C (HCV) are eligible only if polymerase chain reaction is negative for HCV RNA. Patients positive for hepatitis C (HCV) are eligible if they undergo treatment per local guidelines